robots.txt

[m0ldy]

The newest version of phpBB doesn't contain a robots.txt? How important is this for SEO? Should I create one? How do I create one?

 

[smirkley]

Robots.txt is important for any website.

I am sure the phpbb community has likely discussed what should be disallowed. You may want to search their forums for good examples.

Also should add your sitemap location to it too for google.

I dont use phpbb, but like I said, every site should have one. And each brand of forum software has differences.

 

[doubt]

The newest version of phpBB doesn't contain a robots.txt? How important is this for SEO? Should I create one?

Would you like important search engines such as Google, Yahoo, etc. to stop indexing your site?
In that case it's important to have it.
Would you like some other not important engines such as Baidu to stop indexing your site?
Including it in robots.txt does nothing, Baidu ignores it.

 

[Skybound]

If the use of robots.txt is to prevent crawlers and spiders - it is no use.
Just come out of a bot attack that basically ripped through our site for about 14 days, making it almost non accessible. Trying to stop some of the bots and spiders via the robots file proved useless. Eventually had to go to htaccess level to prevent some of the bots.

There is a nice add on for phpBB which called ManageBots, which will add around 350 known bots to your phpBB 'Bots' group. This at least allows you to see which bots are accessing your site.