Hu guys,

How many of you do use Robots.txt file to give instructions to SE spiders what to read in your web site? I think that it is importat at least for Google. What about you?

Is it important?

In my opinion, it is only important when you have pages you don't want the Search Engine robots to find, fetch and/or index.

These days, this is also useful to prevent the Search Engine bots from processing the same content via different URLs i.e. duplicate content. For example, the robots.txt file here @ GIDForums disallows pages like showthread.php and forumdisplay.php among others.

Noindex and nofollow tags are the best tools doing things you are looking for.

I use it to stop google and other serious search engines to consume too much of my bandwidth on folders I'm not interested, for example, downloads, images, memebrs, etc.

I say serious because some rog bots they just dont obey orders....