User-agent field in robots.txt

JdS · #1 01-Jul-2004, 07:28

I have read and re-read the robots.txt exclusion protocol here: http://www.robotstxt.org/wc/norobots.html

This bit in the page is confusing to me...

Quote:

...If more than one User-agent field is present the record describes an identical access policy for more than one robot. At least one field needs to be present per record...

Am I correct in assuming the following example is valid?

Code:

User-agent: googlebot
User-agent: gidbot
User-agent: msnbot
Disallow: /ads/

User-agent: *
Disallow:

In other words, the first three bots should NOT be fetching any pages off the /ads/ folder?

Div · #2 01-Jul-2004, 09:32

Quote:

User-agent: *
Disallow:

Doesn't this contradict the first rule?

JdS · #3 01-Jul-2004, 09:49

Should it? From what I understand, every other bot has access to ALL the pages / folders off the site. The three mentioned above that 'rule', may not access anything on /ads/.

Div · #4 01-Jul-2004, 09:55

So it is understood, you don't have to put it in right?

JdS · #5 01-Jul-2004, 10:12

Very TRUE!

My, you're sharp today...

Just so that we can move forward with this issue; let's say the example robots.txt is like this instead:

Code:

User-agent: googlebot
User-agent: gidbot
User-agent: msnbot
Disallow: /ads/

User-agent: *
Disallow: /ads2/

Div · #6 01-Jul-2004, 11:44

I have looked through a bunch of examples and tutorials. I believe that if you have multiple user agents being disallowed access to a portion of the site, you have to have individual entries for each user agent.
For, eg.

Code:

User-agent: googlebot
Disallow: /ads/

User-agent: gidbot
Disallow: /ads/

User-agent: msnbot
Disallow: /ads/

But, if you have multiple directories being disallowed, you can put them together for a single user-agent. So,

Code:

User agent: *
Disallow: /ads2/
Disallow: /cgi-bin/
Disallow: /gfx/

JdS · #7 20-Jul-2004, 07:13

An update (for anyone else researching the same thing), here's a quote off the email I received from Mr. Martijn Koster:

Quote:

#1 is valid according to that document, yes.

#2 is wrong -- the '*' record is only used if there are no others more suitable.

#3 is correct

#4 is correct

Darksat · #8 02-Sep-2004, 11:19

Can you just have
User-agent: googlebot
Disallow: /ads/

without having to wory about

User-agent: *
Disallow:

?

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Rate This Thread
Rate This Thread:

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
c++/cgi program can't open file in user dir	mikenowo	Apache Web Server Forum	2	04-Mar-2004 20:33
Need Help on checking user input	hihellochao	C Programming Language	5	27-Feb-2004 14:30
Grouping data from MySQL with PHP - Newbie question.	giobbi	MySQL / PHP Forum	12	27-Feb-2004 01:34
text field problm	zuzupus	Web Design Forum	3	04-Sep-2003 06:18
wannabrowswer - http user agent spoofing	jrobbio	Web Design Forum	2	22-Mar-2003 08:11