What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 128 Stack Exchange communities.

The purpose of this question is not to find out how SO protects its sitemap in order to subvert it, so to that end, I don't expect an exact answer.

However, questions around SO's sitemap have popped up:

Stack Overflow Sitemap: WTF?

Does stackoverflow have a sitemap?

Given the initial statement (but hey, an exact answer would be accepted as well), is there any guidance or set of rules/experiences/info which would indicate what to look at when protecting a sitemap like SO's, dos and don'ts as well as the tradeoffs/risks involved?

share|improve this question
1  
Just a quote from Jeff at meta.stackexchange.com/questions/19805/… "If you do anything with the sitemap we don't like, your IP will be banned without hesitation." –  Arjan Dec 28 '09 at 9:41
add comment

1 Answer 1

up vote 13 down vote accepted

We have to protect the sitemap because it's enormous; we only publish the last 50k questions, but that's a huge XML file. It's not like it contains any secrets or anything -- it's just an XML file with a list of the last 50k questions to be updated on any of the trilogy sites.

Before we did this, it used up many gigabytes of bandwidth through incorrect retrievals. We think badly written Firefox plugins were mostly to blame, but it's hard to tell.

Anyway, we use a whitelist type approach. If you're on the whitelist, you get to retrieve the sitemap. If you are not, you don't.

share|improve this answer
3  
oops I just switched my user agent to googlebot, and had a look, it is one scary big document, no wonder you are protecting it. –  waffles Dec 28 '09 at 5:08
    
@Jeff Attwood: I appreciate the info. When you say "whitelist" approproach, can you be more specific? Is it user agent, ip address, both, neither, a combo of things? I'm asking because I'm looking at creating a dynamic sitemap for a site which can become enormous quite fast, and I want to not run into bandwidth issues (basically what I said above about best practices, risks/tradeoffs, etc, etc). Thanks. –  casperOne Dec 28 '09 at 5:22
4  
@casperOne If you are starting up a new site "which can become enormous", I would say that the Sitemap question is probably the one thing that you least should worry about at this time. Focus on getting there first. Protecting the Sitemap file will be a no brainer when you are "enormous". –  Magnus Dec 28 '09 at 9:12
1  
"Before we did this, it used up many gigabytes of bandwidth..." -- "it" == Google? search engines in general? –  Ether Dec 28 '09 at 17:52
    
like I said, we think it was badly written Firefox plugins, so just average users. –  Jeff Atwood Dec 28 '09 at 21:31
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .