11

The purpose of this question is not to find out how SO protects its sitemap in order to subvert it, so to that end, I don't expect an exact answer.

However, questions around SO's sitemap have popped up:

Stack Overflow Sitemap: WTF?

Does Stack Overflow have a sitemap?

Given the initial statement (but hey, an exact answer would be accepted as well), is there any guidance or set of rules/experiences/info which would indicate what to look at when protecting a sitemap like SO's, dos and don'ts as well as the tradeoffs/risks involved?

Improve this question
1
20

We have to protect the sitemap because it's enormous; we only publish the last 50k questions, but that's a huge XML file. It's not like it contains any secrets or anything -- it's just an XML file with a list of the last 50k questions to be updated on any of the trilogy sites.

Before we did this, it used up many gigabytes of bandwidth through incorrect retrievals. We think badly written Firefox plugins were mostly to blame, but it's hard to tell.

Anyway, we use a whitelist type approach. If you're on the whitelist, you get to retrieve the sitemap. If you are not, you don't.

Improve this answer
12
  • 8
    oops I just switched my user agent to googlebot, and had a look, it is one scary big document, no wonder you are protecting it.
    – waffles
    Dec 28 '09 at 5:08
  • @Jeff Attwood: I appreciate the info. When you say "whitelist" approproach, can you be more specific? Is it user agent, ip address, both, neither, a combo of things? I'm asking because I'm looking at creating a dynamic sitemap for a site which can become enormous quite fast, and I want to not run into bandwidth issues (basically what I said above about best practices, risks/tradeoffs, etc, etc). Thanks.
    – casperOne
    Dec 28 '09 at 5:22
  • 4
    @casperOne If you are starting up a new site "which can become enormous", I would say that the Sitemap question is probably the one thing that you least should worry about at this time. Focus on getting there first. Protecting the Sitemap file will be a no brainer when you are "enormous".
    – Magnus
    Dec 28 '09 at 9:12
  • 1
    "Before we did this, it used up many gigabytes of bandwidth..." -- "it" == Google? search engines in general?
    – Ether
    Dec 28 '09 at 17:52
  • like I said, we think it was badly written Firefox plugins, so just average users.
    – Jeff Atwood
    Dec 28 '09 at 21:31
  • @JeffAtwood : but why doing this ? Even googles sitemaps are public. You can definitely guess they are bigger than stack exchange ones.
    – user2284570
    Apr 11 '16 at 22:35
  • @user2284570 that is why.
    – pinkpanther
    Jun 1 '16 at 19:09
  • @pinkpanther : why what ?
    – user2284570
    Jun 7 '16 at 12:10
  • @user2284570 I mean, when we can definitely guess that they are bigger than stackexchange's we can guess that they can afford to make them public, they might have more resources..
    – pinkpanther
    Jun 7 '16 at 15:23
  • 1
    @pinkpanther : but for larger files. This doesn’t explain stack exchange situation that much.
    – user2284570
    Jun 7 '16 at 15:46
  • @JeffAtwood - Many people will suspect that Google sitemaps are enormous due to the size of the Google site to begin with however that's not entirely accurate. A quick check of Google's sitemap file shows that it links to 12 separate sitemaps, none of which are particularly large (from a sitemap perspective) as even the largest one has less than a few hundred records listed. Where this makes sense is that the vast majority of Google's content comes from search listings and content targeted at authenticated users which is not indexable by other search engines.
    – Chris Rutherfurd
    Feb 12 '17 at 23:06
  • 1
    @casperOne - A basic guide to protecting your sitemap would be to restrict access to it based on user agent to known genuine search engine crawlers. I would recommend against IP address based restrictions as the IP's used by crawlers change frequently and many search engines don't even provide the IP ranges of their crawlers, but all legitimate crawlers do have a user agent string that identifies them as a crawler if it is for indexing purposes (which is the only sort of crawler that should be accessing your sitemap file).
    – Chris Rutherfurd
    Feb 12 '17 at 23:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .