What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

SO's Robots.txt file contains the following lines:

#
# disallow adsense bot, as we no longer do adsense. Sorry adsense, 
# nothing personal, but you suck.
#
User-agent: Mediapartners-Google*
Disallow: /

(I was looking for suitable examples to add to my own site)

All well and good - however when using Google's Webmaster tools to check Robots.txt settings, I recieved the following:

Test results

Url         | Googlebot                    | Mediapartners-Google
/           | Allowed by line 2: Allow: /  | Allowed by line 2: Allow: /
/questions | Allowed by line 2: Allow: /  | Allowed by line 2: Allow: /

Changing the Robots.txt to read:

User-agent: Mediapartners-Google
Disallow: /

I get what I expect you actually wanted:

Test results

Url         | Googlebot                    | Mediapartners-Google
/           | Allowed by line 2: Allow: /  | Blocked by line 19: Disallow: /
/questions  | Allowed by line 2: Allow: /  | Blocked by line 19: Disallow: /

Or is it that the Mediapartners spider treats the user-agent more flexibly than the test tool?

share|improve this question

1 Answer 1

main trafic source is a search engines - do you still think Robots.txt file contains errors?

share|improve this answer
4  
I believe Zhaph is talking about the Mediapartners bot, which is the bot used to index the page to decide what ads to serve if Adsense is being used. In this case, SO is trying to block Mediapartners because it's using bandwidth but not doing anything, since SO doesn't use Adsense. –  waiwai933 Sep 5 '09 at 4:11
2  
@waiwai933 - yep, that's right, I was talking about the fact that I didn't think Mediapartners was being blocked. –  Zhaph - Ben Duguid Sep 5 '09 at 16:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .