A simple Google search (filetype:xml site:stackoverflow.com
) tells me the SO sitemap.xml
file is here:
http://stackoverflow.com/sitemap.xml
But When I access this file in the browser, I get a "Page Not Found".
I tried spoofing a Googlebot:
curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" http://stackoverflow.com/sitemap.xml
but I still get the "Page Not Found". Probably SO verifies the Googlebot
How can I still get the SO sitemap.xml
(And other sitemap files linked from it.)