Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 158 Stack Exchange communities.

What is meta?
Here's how it works:
  1. Any Stack Exchange user can ask a question
  2. The community provides support, votes on ideas, and reports bugs
  3. Your voice helps shape the way Stack Exchange operates

A simple Google search (filetype:xml tells me the SO sitemap.xml file is here:

But When I access this file in the browser, I get a "Page Not Found".

I tried spoofing a Googlebot:

curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +"

but I still get the "Page Not Found". Probably SO verifies the Googlebot

How can I still get the SO sitemap.xml (And other sitemap files linked from it.)

share|improve this question

As per the answer from Jeff to the question Why does the Stack Overflow sitemap.xml use a user-agent whitelist instead of a blacklist?:

I agree that checking the user-agent wasn't enough. I added a reverse DNS check as well.

So the short answer is, you can't.

The long answer is, if you have a valid reason to be accessing it - lets imagine you work for an up and coming search engine called Newggle ;) - you could add that to your question or get in touch using the Contact Us link at the bottom of the page if nobody from Stack Exchange spots your question.

share|improve this answer
I guess you could cURL the Google search cache: – Danny Beckett May 17 '13 at 12:54
@DannyBeckett: yes that could be a choice, thanks! – Gabriel Petrovay May 17 '13 at 13:00
@Rob: I am only trying to learn sitemap techniques by example. So in this case I will pick similar a similar website that doesn't hide them – Gabriel Petrovay May 17 '13 at 13:02
@Gabriel It's not that Stack Overflow wants to hide their sitemap per se. It's that it's very traffic-intensive, and they don't want people's requests for the sitemap to negatively impact their traffic. So I don't think they'd mind you getting the sitemap from Google cache. – Chris Jester-Young Dec 13 '14 at 14:46
@Rob : but why doing this ? Even google sitemap is public. – user2284570 Apr 11 at 22:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .