What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

A simple Google search (filetype:xml site:stackoverflow.com) tells me the SO sitemap.xml file is here:

http://stackoverflow.com/sitemap.xml

But When I access this file in the browser, I get a "Page Not Found".

I tried spoofing a Googlebot:

curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" http://stackoverflow.com/sitemap.xml

but I still get the "Page Not Found". Probably SO verifies the Googlebot

How can I still get the SO sitemap.xml (And other sitemap files linked from it.)

share|improve this question

1 Answer 1

As per the answer from Jeff to the question Why does the Stack Overflow sitemap.xml use a user-agent whitelist instead of a blacklist?:

I agree that checking the user-agent wasn't enough. I added a reverse DNS check as well.

So the short answer is, you can't.

The long answer is, if you have a valid reason to be accessing it - lets imagine you work for an up and coming search engine called Newggle ;) - you could add that to your question or get in touch using the Contact Us link at the bottom of the page if nobody from Stack Exchange spots your question.

share|improve this answer
    
I guess you could cURL the Google search cache:http://stackoverflow.com/sitemap.xml –  Danny Beckett May 17 '13 at 12:54
    
@DannyBeckett: yes that could be a choice, thanks! –  Gabriel Petrovay May 17 '13 at 13:00
    
@Rob: I am only trying to learn sitemap techniques by example. So in this case I will pick similar a similar website that doesn't hide them –  Gabriel Petrovay May 17 '13 at 13:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .