71

As you probably know, it’s time for the quarterly publication of our data (affectionately known as the “data dumps”). The process of copying, uploading, and publishing the data over a notoriously fickle network connection is scheduled to begin on Sunday, September 3rd. The process normally takes several days to complete, and historically, at least a few of those data dumps won’t upload successfully. Since this process is starting on a Sunday, and (for added fun) Monday is a US Holiday, some segments of the data we publish may be delayed.

When we return to the office on Tuesday, September 5th, we’ll check the status and take whatever corrective action is necessary (if any). Of course, if the process runs as scheduled and has no hiccups, the data will post as expected, but in the interest of keeping you fully informed, we’re letting you know ahead of time that there may be a delay and why. We’re working to improve this pipeline and make it more robust for the future, so that it doesn’t need this level of proactive monitoring.

6
  • 34
    Thank you for the transparency, and congrats in advance on the Meta Stack Exchange reputation points :-) Aug 31, 2023 at 20:48
  • 4
    We're encountering a few issues with the upload (as anticipated), which we're attempting to resolve as soon as possible. New estimated completion date & time still pending; will update when we know more.
    – Slate StaffMod
    Sep 5, 2023 at 15:37
  • Happy to see y'all being proactive here! It still appears stuck. Any update? Sep 7, 2023 at 17:06
  • 6
    We're currently working with the Internet Archive to mitigate the volatility of us uploading files one by one to their system, which will make our end of the process much tidier and stable. There may be an additional couple days of wait as we explore these options. Apologies for the delay.
    – Slate StaffMod
    Sep 7, 2023 at 20:08
  • 1
    I half jokingly wonder if you could have someone drop by their offices with a hard disk... :D Sep 12, 2023 at 7:01
  • @JourneymanGeek Hide it in a book. They definitely accept those.
    – wizzwizz4
    Sep 12, 2023 at 17:09

1 Answer 1

31

We completed uploading files for the September data dump over the weekend.

All of Archive's background tasks have completed overnight. I now consider the dump complete (including the torrent file) and have updated the main page to indicate yesterday's date (confession: I updated that yesterday in an attempt to trigger a regeneration of the metadata files).


Moving forward

While we weren't able to get to implementation phase this time, we are still working with Archive on a very promising solution that would make the process cleaner on our side, and should make the overall process quicker. The problem we have today is that uploading directly to Archive is not fast to start with (averaging 5mb/s) and, on top of that, we are subject to rate limiting - just like everyone else - that can halt our progress for hours. Then further background processing, including scanning every file for security reasons.

Note: They have throttling and rate limiting and security checking for very good reasons.

The solution

Archive will pull files from us, and they want to work with us on this better solution. It takes us 40 minutes to generate and upload the files, but multiple days to upload them to Archive, and then sometimes multiple days for their processing. If they can pull from us directly into their network without the rate limiting or bandwidth throttling, that's one less thing for us to worry about, and should shorten the process by days.

The downside

While being less hands-on with the data dump certainly frees up our time and headaches, and should get the files to you sooner, we'll gravitate away from being controllers and closer to being consumers. It means we will be even less capable of proactively telling you when the process is complete (e.g. this request).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .