12

Feature request: When a Stack Exchange website is shut down, please save all its pages on https://web.archive.org/ (I don't mean as an XML dump, but each webpage)

E.g., on https://web.archive.org/web/20160711182720/http://productivity.stackexchange.com/ I see that the following questions weren't archived:

This should include not only the homepage and posts, but also user pages and revision histories (so all the attributions for post edits are saved too).


Replies to comments:

Do they actually provide any kind of API or tooling to do this? I've only ever heard of being able to archive single pages manually. We can't possibly do this if they don't have support for tooling to do it. – animuson 5 mins ago

Yes, e.g.:

10
  • 2
    Do they actually provide any kind of API or tooling to do this? I've only ever heard of being able to archive single pages manually. We can't possibly do this if they don't have support for tooling to do it.
    – animuson StaffMod
    Jul 20 at 22:46
  • 4
    @animuson I think they have Archive-it, which serves this purpose. Jul 20 at 22:52
  • 1
  • That Python library is for uploading files (such as our data dump) and not requesting pages to be archived the normal way you desire. And all the APIs there are read-only search options. Archive-It looks like the only solution there. But as far as I can tell, that would still be stored in a completely different location and not in the Wayback Machine.
    – animuson StaffMod
    Jul 20 at 23:02
  • 1
    @animuson, a snapshot at archive-it cost ~500-2000 (See: archive-it.org/blog/products-and-services archive-it.org) the pro package regularly visits and mirrors live sites, for a much greater cost. The related WayBack is only for individuals. There is an API and management tools in the pro package, see first link.
    – Rob
    Jul 20 at 23:08
  • 1
    @animuson Each URL can be saved with an individual GET to https://web.archive.org/save/<URL>. If you are currently viewing a page, you can use the following bookmarklet: javascript:void(window.open('https://web.archive.org/save/'+location.href)) They may have something where you can make bulk requests, or a request to have a site crawled. ... and I see you've already been pointed at their service to do that. IIRC, it used to be easier.
    – Makyen
    Jul 20 at 23:27
  • 2
    @Makyen I am aware of that, and I highly doubt they would approve of us sending tens of thousands of requests via that method, in the same way our servers don't appreciate users hammering our sites with archive requests in a short period. Which is why I asked if there was any API for more ethically achieving this.
    – animuson StaffMod
    Jul 20 at 23:32
  • 3
    @animuson Yep. I should have finished reading prior to posting a comment. I'm sorry about that. I was also operating based on my recollection of what seems to be old information, or, at least, my memory of what they have available is substantially different from what's there now. As you've no doubt already determined, archive.org now has a range of paid offerings ranging from a subscription service to archiving a single copy of a site over a period of 30 days.
    – Makyen
    Jul 20 at 23:42
  • 1
    You might be interested in this request of mine: meta.stackexchange.com/questions/370774/… Oct 12 at 1:49
  • @SonictheAnonymousHedgehog thanks, +1, good idea! Oct 12 at 1:51

You must log in to answer this question.

Browse other questions tagged .