robots.txt

m0ldy

Participant
Joined
Jan 1, 2016
Messages
68
The newest version of phpBB doesn't contain a robots.txt? How important is this for SEO? Should I create one? How do I create one?
 

smirkley

ID'mazing
Joined
Nov 15, 2004
Messages
1,167
Robots.txt is important for any website.

I am sure the phpbb community has likely discussed what should be disallowed. You may want to search their forums for good examples.

Also should add your sitemap location to it too for google.

I dont use phpbb, but like I said, every site should have one. And each brand of forum software has differences.
 

doubt

Tazmanian
Joined
Feb 25, 2013
Messages
4,874
The newest version of phpBB doesn't contain a robots.txt? How important is this for SEO? Should I create one?
Would you like important search engines such as Google, Yahoo, etc. to stop indexing your site?
In that case it's important to have it.
Would you like some other not important engines such as Baidu to stop indexing your site?
Including it in robots.txt does nothing, Baidu ignores it.
 

Skybound

Participant
Joined
Sep 26, 2011
Messages
53
If the use of robots.txt is to prevent crawlers and spiders - it is no use.
Just come out of a bot attack that basically ripped through our site for about 14 days, making it almost non accessible. Trying to stop some of the bots and spiders via the robots file proved useless. Eventually had to go to htaccess level to prevent some of the bots.

There is a nice add on for phpBB which called ManageBots, which will add around 350 known bots to your phpBB 'Bots' group. This at least allows you to see which bots are accessing your site.
 
Top