19

I recently tried to find out why Google doesn't index a question I asked on Bioinformatics (in beta). For fun, I tried archiving it on web.archive.org, but to my surprise SE blocked the request with Reason: Request rate. Is that on purpose?

Is there a reason why questions shouldn't be archived? The content is Creative Commons licensed. If there's something interesting for historians in 50 years, why not?

Here's a screenshot of what I get here https://web.archive.org/web/20211019044811/http://bioinformatics.stackexchange.com/

Enter image description here

@Glorfindel had the great idea of comparing what's in the archive between different sites. It turns out that Ask Different never had a single 4xx problem (yellow), whereas Bioinformatics almost always does:

Enter image description here

Enter image description here

8
  • Did you try waiting and trying again? If that didn't work, did you try contacting [email protected] as the message suggests? Commented Oct 19, 2021 at 19:35
  • 3
    Your IP address is in that picture, are you okay with that?
    – bobble
    Commented Oct 19, 2021 at 19:36
  • 2
    It's interesting to compare the overviews for several sites: https://web.archive.org/web/*/https://3dprinting.stackexchange.com/ https://web.archive.org/web/*/https://apple.stackexchange.com/ https://web.archive.org/web/*/https://bioinformatics.stackexchange.com/ https://web.archive.org/web/*/https://chess.stackexchange.com/ Bioinformatics has a lot of 'not founds' this year ...
    – Glorfindel Mod
    Commented Oct 19, 2021 at 19:36
  • 7
    @bobble it's likely to be the IP address of one of the Wayback Machine servers.
    – Glorfindel Mod
    Commented Oct 19, 2021 at 19:36
  • 2
    @RobertLongson Yes, I have waited, and I have emailed, but I still thought why not ask the question because this is very different from a general block for an end user. In this case it's the archiver that's blocked, not me. That's why I asked here, it's a specifically interesting case of blocking. Commented Oct 19, 2021 at 19:38
  • The rate limiter applies to the entire network of SE sites, in aggregate. Once someone makes too many requests on one or more SE sites put together, access to all sites is restricted. Commented Oct 19, 2021 at 19:39
  • 4
    @bobble It's the IP of the web archive server, not mine. But thanks for asking! Commented Oct 19, 2021 at 19:39
  • 1
    Related: meta.stackexchange.com/q/320850/323179
    – Laurel
    Commented Oct 20, 2021 at 13:39

1 Answer 1

8

One of the ways that Stack Exchange defends itself from Denial of Service and spam attacks is by limiting the number of requests from a single source. This is usually only temporary as the message says.

Anyone anywhere on the internet can press that button and one of the Wayback servers will try to contact Stack Exchange's servers to crawl that page. The Stack Exchange defences probably don't treat Wayback server specially so they see it as making an unusual number of requests and block it.

All we can advise you to do on Meta is follow the instructions and contact [email protected] to see whether they will adjust the rate limiter for this case.

4
  • 2
    I have already written an email. The pattern of bioinformatics vs apple is quite interesting though, isn't it? Bioinformatics is almost always blocked, Apple never. Commented Oct 19, 2021 at 19:44
  • 2
    The only other thing you need is patience then. Normal response time is 6-8 somethings. What you're seeing is likely Wayback's autocrawling rather than the manual crawl you kicked off, it's quite possible they work differently. Commented Oct 19, 2021 at 19:46
  • 3
    Have you seen this image? The crawl almost always goes 4xx: i.sstatic.net/I4nr8.png The discrepancy to Apple is huge. Am I missing something? Commented Oct 19, 2021 at 20:40
  • 2
    Strange thing is (and take it with a pinch of salt - since I don't work here) - I'm half certain they all get served by the same servers. Commented Oct 20, 2021 at 8:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .