17

A simple Google search (filetype:xml site:stackoverflow.com) tells me the SO sitemap.xml file is here:

http://stackoverflow.com/sitemap.xml

But When I access this file in the browser, I get a "Page Not Found".

I tried spoofing a Googlebot:

curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" http://stackoverflow.com/sitemap.xml

but I still get the "Page Not Found". Probably SO verifies the Googlebot

How can I still get the SO sitemap.xml (And other sitemap files linked from it.)

|improve this question
11

As per the answer from Jeff to the question Why does the Stack Overflow sitemap.xml use a user-agent whitelist instead of a blacklist?:

I agree that checking the user-agent wasn't enough. I added a reverse DNS check as well.

So the short answer is, you can't.

The long answer is, if you have a valid reason to be accessing it - lets imagine you work for an up and coming search engine called Newggle ;) - you could add that to your question or get in touch using the Contact Us link at the bottom of the page if nobody from Stack Exchange spots your question.

|improve this answer
  • 3
    I guess you could cURL the Google search cache:http://stackoverflow.com/sitemap.xml – Danny Beckett May 17 '13 at 12:54
  • @DannyBeckett: yes that could be a choice, thanks! – Gabriel Petrovay May 17 '13 at 13:00
  • 1
    @Rob: I am only trying to learn sitemap techniques by example. So in this case I will pick similar a similar website that doesn't hide them – Gabriel Petrovay May 17 '13 at 13:02
  • @Gabriel It's not that Stack Overflow wants to hide their sitemap per se. It's that it's very traffic-intensive, and they don't want people's requests for the sitemap to negatively impact their traffic. So I don't think they'd mind you getting the sitemap from Google cache. – Chris Jester-Young Dec 13 '14 at 14:46
  • 2
    @Rob : but why doing this ? Even google sitemap is public. – user2284570 Apr 11 '16 at 22:32
8

As seems, Stackoverflow blocked users to access their sitemap but I figure it a way to get access to there sitemaps(and hope to not delete my account :) ) Here is my solution:

Page not eligible for rich results known by this test

But when you click on View Rendered HTML, You will see the sitemap contents.

|improve this answer
  • I'm sure there's a reason for SE not to want any stranger off the Internet to have direct access to those files. I wouldn't be posting smileys if I were you. Still, I suppose you have discovered a breach of security. – Mari-Lou A Feb 25 '19 at 12:31
  • 3
    @Mari-LouA I don't think it is a breach of security, at least not on the site of SE. The info in sitemap.xml is normally public, only SE has chosen to only allow the google bot (which is the only tool that should be interested in these sitemap.xml files) access these files for performance reasons. The page the OP is using is intended to test how the Google bot accesses a certain page. If it should do that from a network that is whitelisted by some is up for debate with Google. All the smiley's are fine. It is a good find. – rene Feb 25 '19 at 13:34
  • 3
    to be complete: this is what you'll find in such file: <url><loc>https://stackoverflow.com/questions/1854/python-what-os-am-i-running-on</loc><lastmod>2019-02-23</lastmod></url> <url><loc>https://stackoverflow.com/questions/2914/how-can-i-detect-if-a-browser-is-blocking-a-popup</loc><lastmod>2019-02-21</lastmod></url> <url><loc>https://stackoverflow.com/questions/3585/create-a-new-ruby-on-rails-application-using-mysql-instead-of-sqlite</loc><lastmod>2019-02-22</lastmod></url> – rene Feb 25 '19 at 13:44
0

Use your browser developer tools to change your user agent to googlebot and refresh page. It always works for me in similar cases.

For chrome: (quote from the following article)

  1. Open Chrome and pull down the “View” menu, go down to “Developer” and select “Developer Tools” to open the developer panel.
  2. Click the three dots button in the far right corner, then select “More Tools” and select “Network Conditions”.
  3. Look for “User Agent” and uncheck the box next to ‘Select Automatically’ to reveal all user agent options in Chrome.
  4. Select an optional user agent from the dropdown menu to activate that user agent in Chrome.

I have a nice article about changing user agent settings. It also has steps for other popular browsers. Change the Browser User Agent in Chrome, Safari, & Firefox

|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .