I read an article today about how some webmasters absentmindedly list their *cough* private *cough* folders on their robots.txt (but not securing it) thinking that it would keep the pages (or images ) away from the search engines.

But malicious surfers can easily view this file (robots.txt) to check out folders that are listed on it!

I didn't even think of that until I finished reading the article!!!

Is ther a way that you can secure it? I have private folders that i list in robots.txt and so I need to know.

You needto play with the permissions a little bit with chmod, but I believe if you do secure the file itself then it doesn't work with the search engines. If you have good access to the server you might be able to make the web server route all traffic to robots.txt to index.html, however this might also obstruct the original objective of the robots.txt file.
Nick

You'd have to put .htaccess password protection on the private folder, and just not mention the folder in robots.txt.

Different permissions on robots won't change anything, as SE robots are the same as websurfers.

Garth Farley

Just chmod it to owner read only

__________________
Andrew

uuuummmmmmm.... The SE is a visitor so you have to have it able to be read by the Guest.

If it was owner read only, the only things you could do with it is FTP it back to your machine, or be physically at the terminal & view it from there.

Garth Farley