Don't Index Here!

[GTB]

Wrong

Search engines do

Only some search engines obey a robot.text file, many other will ignore it

Search engines do

No, only some search engines obey a robot.text file, google, msn and yahoo does. I know they do. But many other will ignore it and you need use rules added to an .htaccess file instead to block all.

 

[mysiteguy]

Wrong

Only some search engines obey a robot.text file, many other will ignore it

Google, Bing, Baidu, Yandex, Voila, Yahoo (Slurp) all do. What are these "many" you speak of? I don't know of any major search engine around the world which doesn't obey it.

 

[Danielx64]

@All those who thinks that phpBB is designed wrong in having a group for bots, I'm afraid your all wrong. As GTB rightly points out, you may want guests to view members profiles but you don't want the bots to view/index those pages as the content may not be useful for a search engine.

You may want to try and learn how to think outside the square (or rather box).

 

[ozzy47]

@All those who thinks that phpBB is designed wrong in having a group for bots, I'm afraid your all wrong. As GTB rightly points out, you may want guests to view members profiles but you don't want the bots to view/index those pages as the content may not be useful for a search engine.

You may want to try and learn how to think outside the square (or rather box).

So your saying it's ok that a bot can index a admin only board on the forum, unless you tell it not to?

 

[Danielx64]

So your saying it's ok that a bot can index a admin only board on the forum, unless you tell it not to?

Common sense prevails here.

Example 1: You allow guests to view members profiles but you block bots from viewing/indexing those pages - That's ok.
Example 2: You allows bots to view the "off-topic" forum but you disallow guests to view that same forum - That's called cloaking (In case that wasn't clear, that's NOT ok).

 

[ozzy47]

Common sense prevails here.

Example 1: You allow guests to view members profiles but you block bots from viewing/indexing those pages - That's ok.
Example 2: You allows bots to view the "off-topic" forum but you disallow guests to view that same forum - That's called cloaking (In case that wasn't clear, that's NOT ok).

That is not what I asked.

You have a forum/node on the site that is restricted to "admins", but from what has been said, even though no other "person" other than admins can see this, bots can, unless you tell them they can't see it, that makes no sense.

 

[Danielx64]

That is not what I asked.

You have a forum/node on the site that is restricted to "admins", but from what has been said, even though no other "person" other than admins can see this, bots can, unless you tell them they can't see it, that makes no sense.

Of course you wouldn't allow bots to view private forums. Like what being said before, you set up the bots group like you would with any other group.

 

[GTB]

Who said "major search egines" only?

Like what being said before, you set up the bots group like you would with any other group.

Exactly, which is what I tried to tell him. That you would need to set both guest and bot groups up to not view that forum. But keep in mind, only bots added to spider list with phpBB use bot permission group, any spider agents not added are going to fall under guest permissions. You're not going to added "every single bot out" there to list in phpBB, so both groups should be setup pretty similar really.

 

[ozzy47]

Of course you wouldn't allow bots to view private forums. Like what being said before, you set up the bots group like you would with any other group.

So I create a forum, and I have to disallow groups from seeing it, rather than allowing groups, meaning it is viewable be all groups by default?

 

[Danielx64]

So I create a forum, and I have to disallow groups from seeing it, rather than allowing groups, meaning it is viewable be all groups by default?

Nope in phpBB everyone is denied by default - meaning that you need to add groups that you want to allow viewing the content. So you make a new forum/mode, and only add permissions for the administrator group then anyone who is in administrator group can see the content but no one else can.

 

[ozzy47]

Nope in phpBB everyone is denied by default - meaning that you need to add groups that you want to allow viewing the content. So you make a new forum/mode, and only add permissions for the administrator group then anyone who is in administrator group can see the content but no one else can.

So then how could a bot index a admin only forum if they can't see it by default?

 

[GTB]

Also there are different predefined roles you can set a group for each forum, you can even create extra custom ones for certain things like I have done.

Think of it this way Ozzy. Because I have bot and guest groups. I can let guests download files but not bots

 

[ozzy47]

Also there are different predefined roles you can set a group for each forum, you can even create extra custom ones for certain things like I have done.

Think of it this way Ozzy. Because I have bot and guest groups. I can let guests download files but not bots

I understand that, but if a bot can't see a forum by default, why would they index something they should not be able to see?

 

[GTB]

So then how could a bot index a admin only forum if they can't see it by default?

If you didn't add the bot group for that forum, they probably can't see it. But you would add all groups for the forum though and then pick a forum role (or custom role you created) for bots group - such as using "NEVER" permission role that disabled all viewing permissions, e.t.c

But you could probably block them that way also by removing the bot group for that forum

 

[ozzy47]

Why would you select all groups to be able to see a admin forum then use a "custom role" to remove some groups? Why not just give admins only the ability to see it?

 

[GTB]

I do it simply because you might end up changing things with certain forums afterwards, so better really to add all groups and use permission roles with them. As you might forget a group isn't even added to a forum

 

[ozzy47]

So the bottom line is that by default, the way the system is designed, bots can not index things they can not see as you alluded to earlier. But they could the way you are doing the permissions, if you forget to remove them.

 

[GTB]

you alluded to earlier

I didn't allude to anything, that was not what was you was saying earlier. You keep moving goal posts each time to try and make yourself correct

 

[mysiteguy]

Who said "major search egines" only?

You are dodging the point, trying spin a falsehood, and "moving the goal posts" yourself.

The major and minor search engines obey robots.txt. Name any search engine, large or small, which does not obey robots.txt?

If it is "many" as you stated, then I assume you have information backing this up? Yet you haven't named a single one.

 

[mysiteguy]

@All those who thinks that phpBB is designed wrong in having a group for bots, I'm afraid your all wrong. As GTB rightly points out, you may want guests to view members profiles but you don't want the bots to view/index those pages as the content may not be useful for a search engine.

How is that situation not handled by robots.txt? You turn on member profiles for guests, and block them in robots.txt. Search engines major and minor obey robots.txt.