robots.txt

H

hugh

Adherent
Joined
Feb 20, 2005
Messages
389
Could I invite people to comment on the following robots.txt regarding it's syntax, the bots I've chosen to block, etc...? I'm torn between the usefulness of trying to limit their numbers and making my data as widely available as possible...

Thanks.

hugh
---
User-agent: *

User-agent: Alexibot
Disallow: /

User-agent: Aqua_Products
Disallow: /

User-agent: BackDoorBot/1.0
Disallow: /

User-agent: Black Hole
Disallow: /

User-agent: BlowFish/1.0
Disallow: /

User-agent: Bookmark search tool
Disallow: /

User-agent: BotALot
Disallow: /

User-agent: BuiltBotTough
Disallow: /

User-agent: Bullseye/1.0
Disallow: /

User-agent: BunnySlippers
Disallow: /

User-agent: Cegbfeieh
Disallow: /

User-agent: CheeseBot
Disallow: /

User-agent: CherryPicker
Disallow: /

User-agent: CherryPickerElite/1.0
Disallow: /

User-agent: CherryPickerSE/1.0
Disallow: /

User-agent: Copernic
Disallow: /

User-agent: CopyRightCheck
Disallow: /

User-agent: Crescent
Disallow: /

User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /

User-agent: Crescent Internet ToolPak HTTPOLE Control v.1.0
Disallow: /

User-agent: DittoSpyder
Disallow: /

User-agent: EmailCollector
Disallow: /

User-agent: EmailSiphon
Disallow: /

User-agent: EmailWolf
Disallow: /

User-agent: Enterprise_Search
Disallow: /

User-agent: Enterprise_Search/1.0
Disallow: /

User-agent: EroCrawler
Disallow: /

User-agent: ExtractorPro
Disallow: /

User-agent: FairAd Client
Disallow: /

User-agent: Flaming AttackBot
Disallow: /

User-agent: Foobot
Disallow: /

User-agent: Gaisbot
Disallow: /

User-agent: GetRight/4.2
Disallow: /

User-agent: Harvest/1.5
Disallow: /

User-agent: Hatena Antenna
Disallow: /

User-agent: InfoNaviRobot
Disallow: /

User-agent: Iron33/1.0.2
Disallow: /

User-agent: JennyBot
Disallow: /

User-agent: Kenjin Spider
Disallow: /

User-agent: Keyword Density/0.9
Disallow: /

User-agent: LNSpiderguy
Disallow: /

User-agent: LexiBot
Disallow: /

User-agent: LinkScan/8.1a Unix
Disallow: /

User-agent: LinkWalker
Disallow: /

User-agent: LinkextractorPro
Disallow: /

User-agent: MIIxpc
Disallow: /

User-agent: MIIxpc/4.2
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: Mata Hari
Disallow: /

User-agent: Microsoft URL Control
Disallow: /

User-agent: Microsoft URL Control - 5.01.4511
Disallow: /

User-agent: Microsoft URL Control - 6.00.8169
Disallow: /

User-agent: Mister PiX
Disallow: /

User-agent: NICErsPRO
Disallow: /

User-agent: NetAnts
Disallow: /

User-agent: NetMechanic
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Openbot
Disallow: /

User-agent: Openfind
Disallow: /

User-agent: Openfind data gathere
Disallow: /

User-agent: Oracle Ultra Search
Disallow: /

User-agent: PerMan
Disallow: /

User-agent: ProPowerBot/2.14
Disallow: /

User-agent: ProWebWalker
Disallow: /

User-agent: Python-urllib
Disallow: /

User-agent: QueryN Metasearch
Disallow: /

User-agent: RMA
Disallow: /

User-agent: Radiation Retriever 1.1
Disallow: /

User-agent: RepoMonkey
Disallow: /

User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: SpankBot
Disallow: /

User-agent: Szukacz/1.4
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: Telesoft
Disallow: /

User-agent: The Intraformant
Disallow: /

User-agent: TheNomad
Disallow: /

User-agent: TightTwatBot
Disallow: /

User-agent: Titan
Disallow: /

User-agent: True_Robot
Disallow: /

User-agent: True_Robot/1.0
Disallow: /

User-agent: URL Control
Disallow: /

User-agent: URL_Spider_Pro
Disallow: /

User-agent: URLy Warning
Disallow: /

User-agent: VCI
Disallow: /

User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /

User-agent: WWW-Collector-E
Disallow: /

User-agent: Web Image Collector
Disallow: /

User-agent: WebAuto
Disallow: /

User-agent: WebBandit
Disallow: /

User-agent: WebBandit/3.50
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: WebEnhancer
Disallow: /

User-agent: WebSauger
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebZip
Disallow: /

User-agent: WebZip/4.0
Disallow: /

User-agent: WebmasterWorld Extractor
Disallow: /

User-agent: WebmasterWorldForumBot
Disallow: /

User-agent: Website Quester
Disallow: /

User-agent: Webster Pro
Disallow: /

User-agent: Wget
Disallow: /

User-agent: Wget/1.5.3
Disallow: /

User-agent: Xenu's
Disallow: /

User-agent: Xenu's Link Sleuth 1.1c
Disallow: /

User-agent: Zeus
Disallow: /

User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /

User-agent: Zeus Link Scout
Disallow: /

User-agent: asterias
Disallow: /

User-agent: b2w/0.1
Disallow: /

User-agent: cosmos
Disallow: /

User-agent: dumbot
Disallow: /

User-agent: es
Disallow: /

User-agent: grub
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: hloader
Disallow: /

User-agent: httplib
Disallow: /

User-agent: humanlinks
Disallow: /

User-agent: larbin
Disallow: /

User-agent: libWeb/clsHTTP
Disallow: /

User-agent: lwp-trivial
Disallow: /

User-agent: lwp-trivial/1.34
Disallow: /

User-agent: moget
Disallow: /

User-agent: moget/2.1
Disallow: /

User-agent: naver
Disallow: /

User-agent: psbot
Disallow: /

User-agent: searchpreview
Disallow: /

User-agent: sootle
Disallow: /

User-agent: spanner
Disallow: /

User-agent: suzuran
Disallow: /

User-agent: toCrawl/UrlDispatcher
Disallow: /

User-agent: turingos
Disallow: /

Disallow: /cgi-bin/
Disallow: /vbulletin/attachment.php
Disallow: /vbulletin/avatar.php
Disallow: /vbulletin/editpost.php
Disallow: /vbulletin/member.php
Disallow: /vbulletin/member2.php
Disallow: /vbulletin/misc.php
Disallow: /vbulletin/moderator.php
Disallow: /vbulletin/newreply.php
Disallow: /vbulletin/newthread.php
Disallow: /vbulletin/online.php
Disallow: /vbulletin/poll.php
Disallow: /vbulletin/postings.php
Disallow: /vbulletin/printthread.php
Disallow: /vbulletin/private.php
Disallow: /vbulletin/private2.php
Disallow: /vbulletin/report.php
Disallow: /vbulletin/search.php
Disallow: /vbulletin/sendtofriend.php
Disallow: /vbulletin/threadrate.php
Disallow: /vbulletin/usercp.php
Disallow: /vbulletin/admincp/
Disallow: /vbulletin/modcp/
Disallow: /vbulletin/images/
Disallow: /vbulletin/sendmessage.php
Disallow: /vbulletin/register.php
Disallow: /vbulletin/subscription.php
Disallow: /vbulletin/memberlist.php
 
S

skyhawk133

Fan
Joined
Jan 13, 2005
Messages
775
I didn't look at every user agent, but I did see 1 problem:
Code: 
User-agent: turingos
Disallow: /

Disallow: /cgi-bin/
Disallow: /vbulletin/attachment.php
Disallow: /vbulletin/avatar.php
Disallow: /vbulletin/editpost.php
Disallow: /vbulletin/member.php
Disallow: /vbulletin/member2.php
Disallow: /vbulletin/misc.php
Disallow: /vbulletin/moderator.php
Disallow: /vbulletin/newreply.php
Disallow: /vbulletin/newthread.php
Disallow: /vbulletin/online.php
Disallow: /vbulletin/poll.php
Disallow: /vbulletin/postings.php
Disallow: /vbulletin/printthread.php
Disallow: /vbulletin/private.php
Disallow: /vbulletin/private2.php
Disallow: /vbulletin/report.php
Disallow: /vbulletin/search.php
Disallow: /vbulletin/sendtofriend.php
Disallow: /vbulletin/threadrate.php
Disallow: /vbulletin/usercp.php
Disallow: /vbulletin/admincp/
Disallow: /vbulletin/modcp/
Disallow: /vbulletin/images/
Disallow: /vbulletin/sendmessage.php
Disallow: /vbulletin/register.php
Disallow: /vbulletin/subscription.php
Disallow: /vbulletin/memberlist.php
Should be:
Code: 
User-agent: turingos
Disallow: /

User-agent: *
Disallow: /cgi-bin/
Disallow: /vbulletin/attachment.php
Disallow: /vbulletin/avatar.php
Disallow: /vbulletin/editpost.php
Disallow: /vbulletin/member.php
Disallow: /vbulletin/member2.php
Disallow: /vbulletin/misc.php
Disallow: /vbulletin/moderator.php
Disallow: /vbulletin/newreply.php
Disallow: /vbulletin/newthread.php
Disallow: /vbulletin/online.php
Disallow: /vbulletin/poll.php
Disallow: /vbulletin/postings.php
Disallow: /vbulletin/printthread.php
Disallow: /vbulletin/private.php
Disallow: /vbulletin/private2.php
Disallow: /vbulletin/report.php
Disallow: /vbulletin/search.php
Disallow: /vbulletin/sendtofriend.php
Disallow: /vbulletin/threadrate.php
Disallow: /vbulletin/usercp.php
Disallow: /vbulletin/admincp/
Disallow: /vbulletin/modcp/
Disallow: /vbulletin/images/
Disallow: /vbulletin/sendmessage.php
Disallow: /vbulletin/register.php
Disallow: /vbulletin/subscription.php
Disallow: /vbulletin/memberlist.php
 
H

hugh

Adherent
Joined
Feb 20, 2005
Messages
389
skyhawk133 said:
I didn't look at every user agent, but I did see 1 problem:
Code: 
User-agent: turingos
Disallow: /
 
Disallow: /cgi-bin/
Disallow: /vbulletin/attachment.php
Disallow: /vbulletin/avatar.php
Disallow: /vbulletin/editpost.php
Disallow: /vbulletin/member.php
Disallow: /vbulletin/member2.php
Disallow: /vbulletin/misc.php
Disallow: /vbulletin/moderator.php
Disallow: /vbulletin/newreply.php
Disallow: /vbulletin/newthread.php
Disallow: /vbulletin/online.php
Disallow: /vbulletin/poll.php
Disallow: /vbulletin/postings.php
Disallow: /vbulletin/printthread.php
Disallow: /vbulletin/private.php
Disallow: /vbulletin/private2.php
Disallow: /vbulletin/report.php
Disallow: /vbulletin/search.php
Disallow: /vbulletin/sendtofriend.php
Disallow: /vbulletin/threadrate.php
Disallow: /vbulletin/usercp.php
Disallow: /vbulletin/admincp/
Disallow: /vbulletin/modcp/
Disallow: /vbulletin/images/
Disallow: /vbulletin/sendmessage.php
Disallow: /vbulletin/register.php
Disallow: /vbulletin/subscription.php
Disallow: /vbulletin/memberlist.php
Should be:
Code: 
User-agent: turingos
Disallow: /
 
User-agent: *
Disallow: /cgi-bin/
Disallow: /vbulletin/attachment.php
Disallow: /vbulletin/avatar.php
Disallow: /vbulletin/editpost.php
Disallow: /vbulletin/member.php
Disallow: /vbulletin/member2.php
Disallow: /vbulletin/misc.php
Disallow: /vbulletin/moderator.php
Disallow: /vbulletin/newreply.php
Disallow: /vbulletin/newthread.php
Disallow: /vbulletin/online.php
Disallow: /vbulletin/poll.php
Disallow: /vbulletin/postings.php
Disallow: /vbulletin/printthread.php
Disallow: /vbulletin/private.php
Disallow: /vbulletin/private2.php
Disallow: /vbulletin/report.php
Disallow: /vbulletin/search.php
Disallow: /vbulletin/sendtofriend.php
Disallow: /vbulletin/threadrate.php
Disallow: /vbulletin/usercp.php
Disallow: /vbulletin/admincp/
Disallow: /vbulletin/modcp/
Disallow: /vbulletin/images/
Disallow: /vbulletin/sendmessage.php
Disallow: /vbulletin/register.php
Disallow: /vbulletin/subscription.php
Disallow: /vbulletin/memberlist.php
Click to expand...
If I make the change you've suggested, shouldn't I also remove the top line?

User-agent: *

hugh
 
H

hugh

Adherent
Joined
Feb 20, 2005
Messages
389
Actually my last draft was pretty flawed so I'd forget it altogether. Anyway I've now ditched trying to be complicated and gone to the bare bones of allowing everything except from a few bots...

hugh

---
User-agent: Alexibot
Disallow: /

User-agent: Aqua_Products
Disallow: /

User-agent: BackDoorBot/1.0
Disallow: /

User-agent: Black Hole
Disallow: /

User-agent: BlowFish/1.0
Disallow: /

User-agent: Bookmark search tool
Disallow: /

User-agent: BotALot
Disallow: /

User-agent: BuiltBotTough
Disallow: /

User-agent: Bullseye/1.0
Disallow: /

User-agent: BunnySlippers
Disallow: /

User-agent: Cegbfeieh
Disallow: /

User-agent: CheeseBot
Disallow: /

User-agent: CherryPicker
Disallow: /

User-agent: CherryPickerElite/1.0
Disallow: /

User-agent: CherryPickerSE/1.0
Disallow: /

User-agent: Copernic
Disallow: /

User-agent: CopyRightCheck
Disallow: /

User-agent: Crescent
Disallow: /

User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /

User-agent: Crescent Internet ToolPak HTTPOLE Control v.1.0
Disallow: /

User-agent: DittoSpyder
Disallow: /

User-agent: EmailCollector
Disallow: /

User-agent: EmailSiphon
Disallow: /

User-agent: EmailWolf
Disallow: /

User-agent: Enterprise_Search
Disallow: /

User-agent: Enterprise_Search/1.0
Disallow: /

User-agent: EroCrawler
Disallow: /

User-agent: ExtractorPro
Disallow: /

User-agent: FairAd Client
Disallow: /

User-agent: Flaming AttackBot
Disallow: /

User-agent: Foobot
Disallow: /

User-agent: Gaisbot
Disallow: /

User-agent: GetRight/4.2
Disallow: /

User-agent: Harvest/1.5
Disallow: /

User-agent: Hatena Antenna
Disallow: /

User-agent: InfoNaviRobot
Disallow: /

User-agent: Iron33/1.0.2
Disallow: /

User-agent: JennyBot
Disallow: /

User-agent: Kenjin Spider
Disallow: /

User-agent: Keyword Density/0.9
Disallow: /

User-agent: LNSpiderguy
Disallow: /

User-agent: LexiBot
Disallow: /

User-agent: LinkScan/8.1a Unix
Disallow: /

User-agent: LinkWalker
Disallow: /

User-agent: LinkextractorPro
Disallow: /

User-agent: MIIxpc
Disallow: /

User-agent: MIIxpc/4.2
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: Mata Hari
Disallow: /

User-agent: Microsoft URL Control
Disallow: /

User-agent: Microsoft URL Control - 5.01.4511
Disallow: /

User-agent: Microsoft URL Control - 6.00.8169
Disallow: /

User-agent: Mister PiX
Disallow: /

User-agent: NICErsPRO
Disallow: /

User-agent: NetAnts
Disallow: /

User-agent: NetMechanic
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Openbot
Disallow: /

User-agent: Openfind
Disallow: /

User-agent: Openfind data gathere
Disallow: /

User-agent: Oracle Ultra Search
Disallow: /

User-agent: PerMan
Disallow: /

User-agent: ProPowerBot/2.14
Disallow: /

User-agent: ProWebWalker
Disallow: /

User-agent: Python-urllib
Disallow: /

User-agent: QueryN Metasearch
Disallow: /

User-agent: RMA
Disallow: /

User-agent: Radiation Retriever 1.1
Disallow: /

User-agent: RepoMonkey
Disallow: /

User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: SpankBot
Disallow: /

User-agent: Szukacz/1.4
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: Telesoft
Disallow: /

User-agent: The Intraformant
Disallow: /

User-agent: TheNomad
Disallow: /

User-agent: TightTwatBot
Disallow: /

User-agent: Titan
Disallow: /

User-agent: True_Robot
Disallow: /

User-agent: True_Robot/1.0
Disallow: /

User-agent: URL Control
Disallow: /

User-agent: URL_Spider_Pro
Disallow: /

User-agent: URLy Warning
Disallow: /

User-agent: VCI
Disallow: /

User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /

User-agent: WWW-Collector-E
Disallow: /

User-agent: Web Image Collector
Disallow: /

User-agent: WebAuto
Disallow: /

User-agent: WebBandit
Disallow: /

User-agent: WebBandit/3.50
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: WebEnhancer
Disallow: /

User-agent: WebSauger
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebZip
Disallow: /

User-agent: WebZip/4.0
Disallow: /

User-agent: WebmasterWorld Extractor
Disallow: /

User-agent: WebmasterWorldForumBot
Disallow: /

User-agent: Website Quester
Disallow: /

User-agent: Webster Pro
Disallow: /

User-agent: Wget
Disallow: /

User-agent: Wget/1.5.3
Disallow: /

User-agent: Xenu's
Disallow: /

User-agent: Xenu's Link Sleuth 1.1c
Disallow: /

User-agent: Zeus
Disallow: /

User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /

User-agent: Zeus Link Scout
Disallow: /

User-agent: asterias
Disallow: /

User-agent: b2w/0.1
Disallow: /

User-agent: cosmos
Disallow: /

User-agent: dumbot
Disallow: /

User-agent: es
Disallow: /

User-agent: grub
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: hloader
Disallow: /

User-agent: httplib
Disallow: /

User-agent: humanlinks
Disallow: /

User-agent: larbin
Disallow: /

User-agent: libWeb/clsHTTP
Disallow: /

User-agent: lwp-trivial
Disallow: /

User-agent: lwp-trivial/1.34
Disallow: /

User-agent: moget
Disallow: /

User-agent: moget/2.1
Disallow: /

User-agent: naver
Disallow: /

User-agent: psbot
Disallow: /

User-agent: searchpreview
Disallow: /

User-agent: sootle
Disallow: /

User-agent: spanner
Disallow: /

User-agent: suzuran
Disallow: /

User-agent: toCrawl/UrlDispatcher
Disallow: /

User-agent: turingos
Disallow: /
 
Graham

Graham

I Like It Here :)
Joined
May 27, 2005
Messages
1,120
I dont know much about this stuff but I am sure I remember reading somewhere that robots take notice of the robots.txt on a voluntary basis.

Good bots like google, MSN etc will take notice of what you put in robots.txt but since the reason you want to ban the others is because they are suspect or downright devious isn't it likely that they will just ignore it and spider you anyway ?
 
H

hugh

Adherent
Joined
Feb 20, 2005
Messages
389
It's a mixed bag, some do lietrally what they want while others behave themselves. If you're concerned about bots ignoring your robots.txt you have to start IP banning them...

hugh
 
PalePhoenix

PalePhoenix

Prince of Dorkness
Joined
Dec 11, 2005
Messages
11,951
A freebie site analyzer suggests that our robots.txt file is not properly constructed (see attached). Can anyone tell me why?
 

Attachments

  • robot.txt
    1.2 KB · Views: 9
mtcrowe

mtcrowe

Enthusiast
Joined
Feb 10, 2006
Messages
141
PalePhoenix said:
A freebie site analyzer suggests that our robots.txt file is not properly constructed (see attached). Can anyone tell me why?
Click to expand...

The space between user-agent and disallow.

change the top to
Code: 
###############################
#
#
#
# robots.txt created 02-14-06 
#
User-agent: *
Disallow:

Michael
 
PalePhoenix

PalePhoenix

Prince of Dorkness
Joined
Dec 11, 2005
Messages
11,951
mtcrowe said:
The space between user-agent and disallow.

change the top to
Code: 
###############################
#
#
#
# robots.txt created 02-14-06 
#
User-agent: *
Disallow:
Click to expand...
I made this change. The analyzer says it's still wrong. Do we think the analyzer's result is just a ploy to get me to join?
 
DChapman

DChapman

Devotee
Joined
May 20, 2004
Messages
2,880
PalePhoenix said:
I made this change. The analyzer says it's still wrong. Do we think the analyzer's result is just a ploy to get me to join?
Click to expand...
Where is your robots.txt on your site? It's not in the normal location.
 
PalePhoenix

PalePhoenix

Prince of Dorkness
Joined
Dec 11, 2005
Messages
11,951
[Zoints]DChapman said:
Where is your robots.txt on your site? It's not in the normal location.
Click to expand...
It's in the root directory. Should it be in the '...com/forums/' as well? I just had it there (replaced it now). Google's now ground to a screeching halt.
 
mtcrowe

mtcrowe

Enthusiast
Joined
Feb 10, 2006
Messages
141
PalePhoenix

PalePhoenix

Prince of Dorkness
Joined
Dec 11, 2005
Messages
11,951
mtcrowe said:
I ran the analyzer with no problems. Here's an example of what I'm using for my robots.txt

I also have the robots meta tag on my index page

Also, the robots.txt file needs to be in your root directory. It will be ignored anyplace else.
Click to expand...
You ran it with no problems for your own site or ours? And, if I'm not mistaken, the ALL meta tag is the exact opposite of what we're telling them to do with each of the Disallows. Google appears back up to its old tricks this morning.
 
mtcrowe

mtcrowe

Enthusiast
Joined
Feb 10, 2006
Messages
141
I ran it with no problems on my robots.txt file that I displayed. And I use the all meta tag since I have no disallows.

It's easier to just password protect the directories you don't want search engines access too then to create a set of disallow rules that some won't even bother to obey.

I tried to add my passworded directories to my robots.txt and found that the number of 401 errors on my server actually increased instead of decreased, so I removed the directories and they went back down.

Michael
 
S

SilverIce

SilverIce
Joined
Jan 29, 2006
Messages
128
Guys may i know that the robot.txt how it actually work and also does it really help your page to get indexed easily ?
 
S

SilverIce

SilverIce
Joined
Jan 29, 2006
Messages
128
Do you mean that place that file together in the folder where the htaccess is in ?

And can anyone please answer my question ? Does the robot.txt helps your page to get indexed faster or more or just for the names only ?
 
DChapman

DChapman

Devotee
Joined
May 20, 2004
Messages
2,880
SilverIce said:
Do you mean that place that file together in the folder where the htaccess is in ?

And can anyone please answer my question ? Does the robot.txt helps your page to get indexed faster or more or just for the names only ?
Click to expand...
Robots.txt is the standard most bots on the internet follow when checking to see what part of a site they are allowed to index. It doesn't get you indexed faster, but it allows you to specify which areas of your site are indexed. For example, I use the following robots.txt for vBulletin forums and "disallow" or stop the bots from accessing all of the listed areas:

Code: 
User-agent: *
Disallow: /admincp/
Disallow: /announcement.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /faq.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /misc.php
Disallow: /modcp/
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /subscriptions.php
Disallow: /threadrate.php
Disallow: /usercp.php

In addition, it can save you bandwidth and server resources that the bots use as they crawl those pages that would not benefit you.
 
You must log in or register to reply here.
Top