Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 158 Stack Exchange communities.

What is meta?
Here's how it works:
  1. Any Stack Exchange user can ask a question
  2. The community provides support, votes on ideas, and reports bugs
  3. Your voice helps shape the way Stack Exchange operates

The purpose of this question is not to find out how SO protects its sitemap in order to subvert it, so to that end, I don't expect an exact answer.

However, questions around SO's sitemap have popped up:

Stack Overflow Sitemap: WTF?

Does stackoverflow have a sitemap?

Given the initial statement (but hey, an exact answer would be accepted as well), is there any guidance or set of rules/experiences/info which would indicate what to look at when protecting a sitemap like SO's, dos and don'ts as well as the tradeoffs/risks involved?

share|improve this question
Just a quote from Jeff at… "If you do anything with the sitemap we don't like, your IP will be banned without hesitation." – Arjan Dec 28 '09 at 9:41
up vote 15 down vote accepted

We have to protect the sitemap because it's enormous; we only publish the last 50k questions, but that's a huge XML file. It's not like it contains any secrets or anything -- it's just an XML file with a list of the last 50k questions to be updated on any of the trilogy sites.

Before we did this, it used up many gigabytes of bandwidth through incorrect retrievals. We think badly written Firefox plugins were mostly to blame, but it's hard to tell.

Anyway, we use a whitelist type approach. If you're on the whitelist, you get to retrieve the sitemap. If you are not, you don't.

share|improve this answer
oops I just switched my user agent to googlebot, and had a look, it is one scary big document, no wonder you are protecting it. – waffles Dec 28 '09 at 5:08
@Jeff Attwood: I appreciate the info. When you say "whitelist" approproach, can you be more specific? Is it user agent, ip address, both, neither, a combo of things? I'm asking because I'm looking at creating a dynamic sitemap for a site which can become enormous quite fast, and I want to not run into bandwidth issues (basically what I said above about best practices, risks/tradeoffs, etc, etc). Thanks. – casperOne Dec 28 '09 at 5:22
@casperOne If you are starting up a new site "which can become enormous", I would say that the Sitemap question is probably the one thing that you least should worry about at this time. Focus on getting there first. Protecting the Sitemap file will be a no brainer when you are "enormous". – Magnus Dec 28 '09 at 9:12
"Before we did this, it used up many gigabytes of bandwidth..." -- "it" == Google? search engines in general? – Ether Dec 28 '09 at 17:52
like I said, we think it was badly written Firefox plugins, so just average users. – Jeff Atwood Dec 28 '09 at 21:31
@JeffAtwood : but why doing this ? Even googles sitemaps are public. You can definitely guess they are bigger than stack exchange ones. – user2284570 Apr 11 at 22:35
@user2284570 that is why. – pinkpanther Jun 1 at 19:09
@pinkpanther : why what ? – user2284570 Jun 7 at 12:10
@user2284570 I mean, when we can definitely guess that they are bigger than stackexchange's we can guess that they can afford to make them public, they might have more resources.. – pinkpanther Jun 7 at 15:23
@pinkpanther : but for larger files. This doesn’t explain stack exchange situation that much. – user2284570 Jun 7 at 15:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .