What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

Usually we tend to organize the sitemaps.xml files in a human way (by categories).

But for Stackoverflow, due to the number of modifications in each categories, all the sitemaps files would need to be regenerated and they would also be downloaded all the time by the search engines.

So do you know how is the sitemap index file of Stackoverflow organized? Is it organized like a "change log" of only new URLs since last sitemap.xml generation?

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <sitemap>
      <loc>http://meta.stackoverflow.com/sitemaps/2010_02_05.xml.gz</loc>
      <lastmod>2010-02-05</lastmod>
   </sitemap>
   <sitemap>
      <loc>http://meta.stackoverflow.com/sitemaps/2010_02_04.xml.gz</loc>
      <lastmod>2010-02-04</lastmod>
   </sitemap>
   ...
</sitemapindex>

This means that:

  • Content update notification is only a matter of "changefreq" and "priority" in previous sitemaps.
  • Content deletion is only a matter of HTTP 404?
share|improve this question
    
Is there a reason you can't download the sitemap itself and find out for yourself? You may have to impersonate the googlebot, but that's relatively easy. –  Adam Davis Feb 5 '10 at 17:26
    
2 reasons: Jeff said in some comment that he does not like it and could ban [my static] IP. and he said to have added some reverse DNS lookup. :) –  Toto Feb 5 '10 at 17:30
    
Why would he dislike it? As long as you are not automatically crawling the site to screen scrap it (as Google does in order to work at all, I love the irony) and hit the server too much you should be OK. –  perbert Feb 5 '10 at 18:21
    
@voyager - Because sitemaps contains a LOT more information than the RSS, and it's huge (up to 10MB and 50k urls), people were slurping it down frequently. But because it is so huge, it consumes a lot of bandwidth, and Jeff decided to restrict it to search engines only. Not only that, but it's dynamic - so it eats up a lot of database and CPU time to generate. –  Adam Davis Feb 5 '10 at 18:42
    
@Polyline: Precisely, get it once and store locally. Be civilized when programming. w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic –  perbert Feb 5 '10 at 19:02

2 Answers 2

up vote 2 down vote accepted

http://www.google.com/search?q=sitemaps.xml+site%3Astackoverflow.com+filetype%3Axml

Click on "Cached" and you'll find the sitemaps file.

It is incomplete, I assume google doesn't cache the entire thing, but it should give you a significant amount of information as to how the whole thing is put together, and some of the tradeoffs made.

You should also check out his blog post on sitemaps. It doesn't give much detail, but it does give a little insight.

share|improve this answer
    
Good point: we should not forget to add the "X-Robots-Tag: NOARCHIVE" directive for the sensitive static xml files. :) –  Toto Feb 5 '10 at 21:21
    
Unfortunately it is difficult to see if this sitemap.xml is cut or a "changelog". But at least I can see that the urls are ordered by changedate DESC –  Toto Feb 5 '10 at 21:46
2  
Found the answer here: meta.stackexchange.com/questions/33965/… They publish the last 50k questions. –  Toto Feb 6 '10 at 16:44

I believe that Jeff might be 404 it not only filtering by User Agent, but also by IP Address, as I could not access it (get 200) by changing my UA to Google's Bog, MS's nor Yahoo's.

It wouldn't surprise me, it's actually a very good idea, as if I were making an illicit crawler, I'd try to spoof the UA to look like Google, but I could not spoof Google's IP Address.

share|improve this answer
    
There is a reverse DNS lookup check and you have to be whitelisted: meta.stackexchange.com/questions/37231/… –  Toto Feb 5 '10 at 21:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .