Site Security Keeping Your Community Safe from Hackers and Other Unwelcome Visitors.

Reply
 
Thread Tools

  #1  
Old 12-08-2005, 09:01 PM
finkey finkey is offline
TAZ Rookie
 
Join Date: Dec 2005
Admin Experience: Beginner
Location: Keaau, HI
Posts: 6
finkey is on a distinguished road
.htaccess and robots.txt
I want to stop prying eyes from looking at my robots.txt like this: http://www.foo.com/robots.txt. It could give them clues about restricted directories. The less they know, the better. If I use .htaccess, will it interfere with robots' ability to crawl as directed around my site? Any better solution?
Reply With Quote
  #2  
Old 12-08-2005, 09:06 PM
DChapman DChapman is offline
Tazmanian
 
Real Name: David
Join Date: May 2004
Posts: 1,439
DChapman is a jewel in the rough
Default
Do you want the directories to be open to everyone? Just not spiderable? Or can access to the specific directories be locked down?

I would very strongly recommend you not attempt to lock down your robots.txt with .htaccess
Reply With Quote
  #3  
Old 12-09-2005, 11:33 AM
finkey finkey is offline
TAZ Rookie
 
Join Date: Dec 2005
Admin Experience: Beginner
Location: Keaau, HI
Posts: 6
finkey is on a distinguished road
Default
The particular sub directories will be password protected with .htaccess. Your answer points to not use file protection on robots.txt. The directory for robots.txt is my home directory and has to be open to everyone. So, there's no way to stop snooping on the robots.txt? Thanks!
Reply With Quote
  #4  
Old 12-09-2005, 03:04 PM
Moparx's Avatar
Moparx Moparx is offline
GNU/Linux Inside
 
Join Date: May 2004
Admin Experience: Advanced
Posts: 675
Moparx will become famous soon enough
Default
there is no way to stop anyone from looking at your robots.txt.

if the subdirectories listed are password protected already why dont you just remove the listings from the file?
__________________
Reply With Quote
  #5  
Old 12-09-2005, 05:41 PM
DChapman DChapman is offline
Tazmanian
 
Real Name: David
Join Date: May 2004
Posts: 1,439
DChapman is a jewel in the rough
Default
Quote:
Originally Posted by finkey
So, there's no way to stop snooping on the robots.txt? Thanks!
Not in a manner that wouldn't hurt you.

If your directories are password protected, just don't list them on robots.txt
Reply With Quote
  #6  
Old 12-10-2005, 09:11 AM
finkey finkey is offline
TAZ Rookie
 
Join Date: Dec 2005
Admin Experience: Beginner
Location: Keaau, HI
Posts: 6
finkey is on a distinguished road
Default
Part of the concern was could robots harvest shopping cart records. I don't plan on allowing customers access to that anyway on secure server. They'll have to print out their invoice at point of sale. I will delete the online record after a day or so.

Thanks for the solution and assist.
Reply With Quote
Reply

  Admin Zone Forums > The Community Zone > Managing an Online Community > Site Security





Currently Active Users Viewing this Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
robots.txt file question - doesn't seem to be working... jilly Servers and Hosting 6 12-25-2005 01:17 PM
Slow SE Indexing of Forum...Robots.txt to blame? stanley Forum SEO 0 11-12-2005 12:32 PM
My robots.txt phoenixdown21 Forum SEO 1 10-25-2005 11:25 AM
Should I put the robots.txt file?? zix vBulletin 3 04-29-2005 12:19 PM
Read this: Robots.txt GetYouPaid Forum SEO 3 03-07-2005 10:30 PM


 

All times are GMT -4. The time now is 03:47 AM.


Powered by: vBulletin
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Page generated in 0.05642295 seconds with 13 queries
The Admin Zone copyright 2003-2014 All Rights Reserved. Content published on The Admin Zone requires permission for reprint.