Is your forum searchable by spiders?

jilly

Fan
Joined
Jan 14, 2004
Messages
929
I have my board set to not be searched by spiders, because the board is a women's support group, and they would not want the personal type things they talk about to show up in a google search on our topic or their username, if someone knew them and tried searching by their 'nickname'.

However I believe that this probably makes the board membership grow much slower than it would. It's a toss up between making it harder for the board to be found, and keeping the community more private..My main website that the boards are associated with is indexed, just not the posts on the board.

I was just curious about other boards:

1. How long has your board been up?
2. How many members do you have?
3. Is your board indexed by search engines?

For me
1. april 2001
2. 6000
3. No
 

Jon

Enthusiast
Joined
Feb 3, 2004
Messages
136
It's been up Since October 03, It has 53 members, (so far) and can be indexed by spiders, in fact you'd be suprised the wierd stuff my board is indexed in the top 30 at Google for. :eek:
 

Ras Masta

Participant
Joined
Feb 13, 2004
Messages
70
My board had been up since Jan 2001 before the crash
It had 1581 Members
Yes it was indexed.
 

Caislander

Adherent
Joined
Jan 15, 2004
Messages
330
10 april 02
8000
yes
ps: quiet anders, generally this opition is setup in your robots.txt file
 

Ted S

Tazmanian Master
Joined
Feb 19, 2004
Messages
6,127
1. Mid 2000
2. 22,000
3. For sure

One important thing to remember about search engine indexing is that by default most forum software does not work well with bots. Yes, robots files are vital to being indexed well but they are not the thing to worry about! While google will crawl your php urls a bit they really don't handle anything with a session or unique key well, if you want to really get good rankings find some hacks to make your dynamic php urls APPEAR to be static .html urls. For vBulletin this is very easy and something I recommend all forums do for their showthread and forumdisplay links if nothing else. The more the engines can index the more obscure queries your board will show up for...
 

Anonymous

Habitué
Joined
Jan 6, 2004
Messages
1,319
Lakecs.com
Feb 2003
163
Yes

RiderInfo.com
Feb 8, 2004
54
yes

I brought lakecs.com online last with vB2 Feb and spend the most effort on making the site search engine spider friendly. This site is not promoted except for local word of mouth.

I brought Riderinfo.com online jan 2004 with Deluxe Portal and had problems. We closed it and started from scratch and lost 37 members. We came back online with vB3 and have built back up. vB3 seems spider friendly out of the box. This site is promoted with go click.
 

CyanSmoker

Beanie and Shades
Joined
Feb 18, 2004
Messages
356
Y-F / Militate.com:
1. I think it's going to be exactly one year...
2. 3626
3. Yes
 

AngelAmidala

Adherent
Joined
Feb 10, 2004
Messages
264
Christian Forums


1) Started November 2001
2) 27,782
3) Yes and No

Everything used to be spidered...but several members were concerned with their information being able to be seen by non-members and people doing searches on places like Google. So we made use of the "hide all information from non-members" feature which keeps non-members from viewing your profile as well as hiding all your threads, posts, etc. from Google. If members enable that, they don't get spidered. (Though there was a bug in this feature and it spidered people anyway. Bug was found and fixed. ;) )
 

Elbonian

Aspirant
Joined
Mar 1, 2004
Messages
36
Non-Christian Forums :D

1) Started in June, 2000 (with UBB)
2) 11,192 registrations (lots of "never returns," though).
Busy hour tends to be 300-450 simultaneous users.
3) Yes (but no "hacks" to increase indexing).

One thing to note about allowing spiders is that they will run up your bandwidth usage. I can see bursts of traffic in our MRTG graph where the spiders seem to be retrieving stuff. If indexing isn't valuable to you, and if you are paying for extra bytes transferred, then you need to change your robots.txt file to exclude spiders in order to cut down your bandwidth costs.
 
Top