Robot.txt File for Vbulletin

OregonAgent

Aspirant
Joined
Jan 30, 2013
Messages
39
I just discovered what a Robot.txt file is and what it does, thanks to esquire for the resource list to familiarize myself with SEO.

Currently, this is how my Robot.txt file reads. This must be the default file because I have never done anything with it.

User-agent: *
Allow: /

User-agent: Googlebot
Allow: /

#Begin Attracta SEO Tools Sitemap. Do not remove
sitemap: http://cdn.attracta.com/sitemap/2613860.xml.gz
#End Attracta SEO Tools Sitemap. Do not remove (does this really mean I should not remove, even though any of the other text files I have seen do not contain anything from Attracta?)

I have been googling examples of Robot.txt files to see what people are doing in Vbulletin. Here is what I found, but I am not sure if this leaves anything remaining to be indexed?

User-agent: AgentLinkus-Google (is this right?)
Disallow:

User-agent: *
Disallow: /go/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /attachment.php
Disallow: /search.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /editpost.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /login.php
Disallow: /subscription.php
Disallow: /private.php
Disallow: /report.php
Disallow: /sendmessage.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /postings.php
Disallow: /sendtofriend.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /showgroups.php
Disallow: /announcement.php

Sitemap: http:///sitemapurl.com (should I use my sitemap from vbseo?

Do you know if there is a way to allow my VIP usergroup members to be indexed, while all other members are not?

Is this too many things to disallow? Seems I might not have anything left?
 
Last edited by a moderator:

esquire

Habitué
Joined
Jul 11, 2010
Messages
1,588
First noindex the pages then put up the robots.txt. If you do this, everything will remain in the index. I think that if you put this in headinclude template with the other meta data (and override any meta robots), this should work.

<vb:if condition="in_array(THIS_SCRIPT, array('tags', 'member', 'memberlist'))">
<META NAME=”ROBOTS” CONTENT=”NOINDEX,FOLLOW”>
</vb:if >

You may wish to exclude admin and VIP profiles. That you can do like this:

<vb if condition="is_member_of($bbuserinfo,1,2,3,4,8,13)" />
//****
</vb:if>
 

BirdOPrey5

#Awesome
Joined
Aug 14, 2008
Messages
4,217
This is what I use for my 4.x forum... it's not perfect but it's a lot better than nothing.

Code:
User-agent: *
Disallow: /admincp/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /modcp/
Disallow: /attachment.php
Disallow: /editpost.php
Disallow: /image.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usernote.php
 

esquire

Habitué
Joined
Jul 11, 2010
Messages
1,588
They do but we're talking about SEO, not dealing with bad bots. I have separate more complex lines to deal with them, scrapers and general spiders that have no business where they are.
 
Top