5

I am interested in being notified when a new data dump is published on the internet archive. I want to find some way to be automatically notified - ideally an RSS feed, but an email or something else that can be followed easily would work. I do not need immediate notification, as in within seconds or minutes. A day or two would also be OK. Maybe a week at most.

I am aware of but not looking for All Stack Exchange data dumps

16
  • 3
    As someone who wants to seed the current datadump (even if I don't have the space/time for the old ones) - there's a certain value in updates of updates so I can dump the old dump and seed the new one Sep 11, 2023 at 11:15
  • 4
    I feel like the "upload is complete" notification would have to come from Archive, since we don't even know when the files are all made available (we know when they've all been uploaded, but before they appear in our directory and available to you, they are subject to processing on their side that has been ... unpredictable). I've been refreshing a page like this.
    – Aaron Bertrand Staff
    Sep 11, 2023 at 11:53
  • 2
    And when these changes Slate mentioned get implemented - hopefully before the December data dump - we'll have even less direct insight into "final state."
    – Aaron Bertrand Staff
    Sep 11, 2023 at 11:57
  • @AaronBertrand yes, something from archive.org would be great. However, I've not found a way to get some sort of update from there. Maybe there is one I don't know of. Or maybe there is some other channel that somebody knows of. It's why I asked the question.
    – VLAZ
    Sep 11, 2023 at 12:00
  • 2
    I'm not sure if anything exists there either (or a channel to ask). But a simple crawler/script could parse this page and notify you when the date changes (which is something we do manually when we determine that the upload is "ready") or this page (searching for {this month 3 letters english}-{this year} which means a new one has at least started). I did this on 9/9 in spite of one file being "parked" - waiting on Archive to determine what file that even is, since the log I have access to is cryptic.
    – Aaron Bertrand Staff
    Sep 11, 2023 at 12:06
  • 3
    One thing we could potentially do, and note that this isn't a promise or a commitment to do it, just floating an idea: When we know that the data dump is ready, we could post here with tags announcement / featured / data-dump, which then userscripts / rss here could help sort you out. This would still be subject to the delay of us knowing and confirming that the dump is ready. And it would require a bit of coordination on our side, but not crazy, that would add to that delay.
    – Aaron Bertrand Staff
    Sep 11, 2023 at 12:14
  • 2
    Posting as a comment since I'm not entirely sure this is an answer: can't you just use the archive.org rss per subject: archive.org/services/…?
    – Erik A
    Sep 11, 2023 at 13:48
  • 3
    @ErikA But I think that demonstrates part of the problem: that feed doesn't (yet) have the newest dump listed.
    – Aaron Bertrand Staff
    Sep 11, 2023 at 14:06
  • @ErikA I tried the archive.org RSS feed but for some reason I wasn't getting any results. I must have used the parameters for it wrong. I was going to look more into it later. Still as Aaron noted - it doesn't have the 09-09 dump yet. I don't know how long it takes to update. If it's, say, ~5 days it might be OK.
    – VLAZ
    Sep 11, 2023 at 14:35
  • Maybe it doesn't show up in the feed because the new dump isn't fully updated yet? The archive page indicates it's "currently being modified/updated", and despite the topic showing "Stack Exchange Data Dump - 2023-09-09", the metadata file in the dump still indicates the old "2023-06-14" date.
    – zcoop98
    Sep 11, 2023 at 22:35
  • 2
    @VLAZ that specific message is a banner that appears on the archive page, I thought only when the metadata file was being updated, but it has been removed now, even though I am still waiting on one file and the metadata file is still showing the June date. I wish I understood all of the mechanics but the archive definitely works in mysterious ways. ¯\_(ツ)_/¯
    – Aaron Bertrand Staff
    Sep 12, 2023 at 11:45
  • 2
    @VLAZ it has returned.
    – Aaron Bertrand Staff
    Sep 12, 2023 at 16:08
  • 2
    Well, as of this morning, this data dump appears complete. I know it's not the automated notification you want, but it's the best I can do for now. :-)
    – Aaron Bertrand Staff
    Sep 13, 2023 at 12:35
  • 2
    @AaronBertrand I will take the Aarontomated notification :P I hope in the future I wouldn't need staff to notify me. The RSS feed is not updated yet but maybe that is also a matter of time. If so, it would be a good enough approach.
    – VLAZ
    Sep 13, 2023 at 13:12
  • 1
    Aarontomation: The current data dump was completed yesterday.
    – Aaron Bertrand Staff
    Mar 6 at 16:19

0

You must log in to answer this question.

Browse other questions tagged .