Questions about /posts/ and others in robots.txt

Lpspider

Enthusiast
Joined
Nov 26, 2005
Messages
145
I run Xenforo.

I'm considering disallowing:

/posts/
/find-new/
/members/

but have not yet.

Two questions:

1. Should these be disallowed?

2. What should I do if a large amount of threads with /posts/ in the URL are indexed by Google?

Surely removing them will have a negative impact, at least short-term, for posts already indexed. They'll be deindexed, reducing any traffic they were bringing in, and possible GWT errors. On the other hand, can I test if they're being treated as duplicate content?

What would you do? I realize blocking /posts/ is probably best practice and should have been implemented some time ago.

3. What should I be looking at in GWT?
 

we_are_borg

Administrator
Joined
Jan 25, 2011
Messages
5,472
1. I don't see any reason to disallow this.

2. Are the links in Google valid does it go where it needs to go.
 

Lpspider

Enthusiast
Joined
Nov 26, 2005
Messages
145
They go where needed, yes. Mostly concerned with duplicate content.
 

GrrasSolutions

Aspirant
Joined
Mar 5, 2019
Messages
39
It depends on you, which file you have to allow and what you have to Disallow, because you know what is important things of the website. and what files you should get crawled by search engine.
 
Top