Stack Overflow Sitemap: WTF?

Question

There have been many questions about the Stack Overflow sitemap, but this question is not like those. My goal is not to figure out how the sitemap works, I just want to look at it (don't worry, I'm not going to abuse it, like the issue here!). I looked at https://stackoverflow.com/robots.txt and found the reference to https://stackoverflow.com/sitemap.xml, but I get a 404 when I try to access the sitemap. How does this work? Can you explain where the sitemap is? I see a comment in robots.txt that that reference to sitemap.xml isn't technically valid - if the real reference is in another place, what is this place? (Also, this question may belong on a different site, please migrate it if so.) Thanks!

UPDATE: From the answers, I've concluded that the sitemap is actually at https://stackoverflow.com/sitemap.xml, but access is restricted to only googlebots. Thanks!

Answer 1

Unless you're Google, Bing, or another valid search engine, why do you need to see it?

http://en.wikipedia.org/wiki/Sitemaps

The Sitemaps protocol allows a webmaster to inform search engines about URLs on a website that are available for crawling. A Sitemap is an XML file that lists the URLs for a site. It allows webmasters to include additional information about each URL: when it was last updated, how often it changes, and how important it is in relation to other URLs in the site. This allows search engines to crawl the site more intelligently. Sitemaps are a URL inclusion protocol and complement robots.txt, a URL exclusion protocol.

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184

Sitemaps are a way to tell Google about pages on your site we might not otherwise discover. In its simplest terms, a XML Sitemap—usually called Sitemap, with a capital S—is a list of the pages on your website. Creating and submitting a Sitemap helps make sure that Google knows about all the pages on your site, including URLs that may not be discoverable by Google's normal crawling process.

Answer 2

There is currently no public-facing site map. The site map is the 6 buttons at the top of the page, your name link to your profile, and the glorious little envelope. There aren't too many dark corners around here...or if there are, nobody's told me about them--which is par for the course.

Answer 3

If my memory serves me correctly, the SO sitemap isn't what most people think of when they think of a sitemap. It's not a couple nested unordered-lists showing links to your profile page, etc. If I remember correctly, it's a massive page containing hundreds of questions used for quick-indexing of SO-content, hence its restricted access.

Answer 4

As can be seen at How does Stack Overflow protect its sitemap?, the Stack Overflow sitemap is enormous and shows the last 50,000 questions on the site, and it is specifically mentioned that it used to use up many gigabytes of bandwidth for incorrect retrievals.

A 404 is returned because of restrictions in place to ensure only legitimate crawlers have access to the file. By and large there is nothing special about the sitemap and no hidden pages on it; it simply provides search engines with the latest 50,000 questions asked and links to those questions. You have access to the same data by just going to the main Stack Overflow page and seeing all of the questions asked there.