Possible delay in Data Dump release - now targeting September 5th

Question

As you probably know, it’s time for the quarterly publication of our data (affectionately known as the “data dumps”). The process of copying, uploading, and publishing the data over a notoriously fickle network connection is scheduled to begin on Sunday, September 3rd. The process normally takes several days to complete, and historically, at least a few of those data dumps won’t upload successfully. Since this process is starting on a Sunday, and (for added fun) Monday is a US Holiday, some segments of the data we publish may be delayed.

When we return to the office on Tuesday, September 5th, we’ll check the status and take whatever corrective action is necessary (if any). Of course, if the process runs as scheduled and has no hiccups, the data will post as expected, but in the interest of keeping you fully informed, we’re letting you know ahead of time that there may be a delay and why. We’re working to improve this pipeline and make it more robust for the future, so that it doesn’t need this level of proactive monitoring.

Answer 1

We completed uploading files for the September data dump over the weekend.

All of Archive's background tasks have completed overnight. I now consider the dump complete (including the torrent file) and have updated the main page to indicate yesterday's date (confession: I updated that yesterday in an attempt to trigger a regeneration of the metadata files).

---

Moving forward

While we weren't able to get to implementation phase this time, we are still working with Archive on a very promising solution that would make the process cleaner on our side, and should make the overall process quicker. The problem we have today is that uploading directly to Archive is not fast to start with (averaging 5mb/s) and, on top of that, we are subject to rate limiting - just like everyone else - that can halt our progress for hours. Then further background processing, including scanning every file for security reasons.

Note: They have throttling and rate limiting and security checking for very good reasons.

The solution

Archive will pull files from us, and they want to work with us on this better solution. It takes us 40 minutes to generate and upload the files, but multiple days to upload them to Archive, and then sometimes multiple days for their processing. If they can pull from us directly into their network without the rate limiting or bandwidth throttling, that's one less thing for us to worry about, and should shorten the process by days.

The downside

While being less hands-on with the data dump certainly frees up our time and headaches, and should get the files to you sooner, we'll gravitate away from being controllers and closer to being consumers. It means we will be even less capable of proactively telling you when the process is complete (e.g. this request).