15

Sometimes an Archive.org capture shows too many requests instead of the site:

https://web.archive.org/web/20181202194435/https://english.stackexchange.com/

I think (but I don’t have a link) that this has also happened for individual questions, which is bad because there might only be one capture of said question. I try to use Archive.org for recovering dead images and it always feels like a slap in the face when I see this.

It’s not possible to give anyone/anything no limit but I think the site should have less limiting.

(Also it should be “an unusually high number” in the error message.)

  • 2
    Given anyone can request a new page be archived at any time, I'm pretty sure just blanket allowing them with no rate limits would never be an option. That just allows them to become a proxy for abuse to our servers. – animuson Dec 23 '18 at 23:47
  • You're imagining DDOS by proxy via the Internet archive?! Hah, now I've heard everything! That's not very plausible. :p – Jeremy Dec 24 '18 at 1:21
  • 2
    It also presumes a binary: disabled or enables rate limiting. Nobody is asking you to allow 10k requests per second, but neither does it make sense to throttle this important service to the same rate as a random user. – Jeremy Dec 24 '18 at 1:23
  • 2
    Is archive.org subject to the same rate limits as other search engines? – 1201ProgramAlarm Dec 24 '18 at 1:49
  • 3
    @Jean This very question did originally ask us not to rate limit them at all. And this service isn't throttled at the same limits as regular users, but it's certainly not given search engine levels of requests either. I don't know what the exact thresholds are for which services, but looking at the logs for the date and time from the screenshot, I'm not at all surprised it got rate limited, and it should have been. We got 878 requests from them during that one minute. – animuson Dec 24 '18 at 3:12

You must log in to answer this question.

Browse other questions tagged .