68

We release our quarterly Data Dump towards the end of every quarter. We begin processing the Data Dump at the beginning of the last month of a quarter, but for the past three quarters, we’ve run into different delays that were outside of our control: the Internet Archive had an outage, rate limiting, etc. While the goal is to automate this process soon, until we’re able to do that, we want to set realistic expectations as to when Data Dumps will be released.

We’re committing to making sure that Data Dumps are released by the last day of the quarter. If processing goes off without any delays, we’ll release them earlier. Because of the holiday season at the end of the year, the Data Dump for Q4 may not be released till the first week of January; however, if it is processed earlier, we will release it in December. You can expect to see Data Dump releases by these dates:

  • March 31, 2024
  • June 30, 2024
  • September 30, 2024
  • January 3, 2025

We’re hoping this new schedule will provide realistic expectations of the release dates. In the unlikely event there is an unexpected delay, we will communicate as early as possible.

Improve this question
10
  • 29
    so... 6-8 weeks? :D
    – Journeyman Geek
    Commented Feb 6 at 15:13
  • 2
    It's not clear what the extent of automation is, but how resilient is the automation to the format and structure of the data dumps? Does it make sense to have conversations about what the dumps contain and how they are structured before going too far down the path of automation? Or would those conversations take too long? Or is the process of automating robust to potential changes in the content and structure of the data dumps and the first step should be to automate the process and free up bandwidth from people who, at some point in the future, may be able to update the structures?
    – Thomas Owens
    Commented Feb 6 at 15:46
  • 16
    @ThomasOwens The content of the files and the automation around getting them to IA are separate and not co-dependent. In other words, we can work on changing the content of the files without affecting the upload automation, and vice versa.
    – Aaron Bertrand Staff
    Commented Feb 6 at 15:48
  • 2
    @AaronBertrand That's awesome news.
    – Thomas Owens
    Commented Feb 6 at 15:50
  • 7
    @ThomasOwens That said, I don't know of any short- or long-term plans to change the files in any way (and I'd be concerned we'd break a lot of people if we did).
    – Aaron Bertrand Staff
    Commented Feb 6 at 15:57
  • @AaronBertrand Agreed. Any changes would have to be discussed with relevant stakeholders and planned out to not break consumers. But after trying to work with the data, there are definitely opportunities to improve it. Not sure how that fits in with other work, though.
    – Thomas Owens
    Commented Feb 6 at 16:10
  • 5
    "We’re committing to making sure that Data Dumps are released by the last day of the quarter" I don't get it: does it mean that the data for Q4 will be released before the end of Q4? Doesn't it make the dump incomplete? I would expect you to release the dump shortly after the end of the quarter, not before it! What am I missing?
    – Fabio says Reinstate Monica
    Commented Feb 8 at 11:15
  • 3
    @Fabio the quarterly dumps have traditionally included data not aligning to calendar quarters, they’ve included data up to the end of an offset month. The March data dump from last year, for example, created on March 1st or so, included data up until the end of February (roughly). This year, the March data dump will still contain data through February, we’re just giving ourselves a longer window to get it to people. The creation of the dump on the first weekend of the last month of the quarter is never the issue; it’s always the struggle to transmit the files to Internet Archive.
    – Aaron Bertrand Staff
    Commented Feb 8 at 11:24
  • @AaronBertrand: Just out of curiosity, do you use the "upload via S3" feature for IA, or do you transfer the files through their website, or some other method? (I've had to create a few IA items, and uploading from EU was a massive struggle... until I realized the CLI did it differently.)
    – user1686
    Commented Feb 8 at 12:53
  • 4
    @user1686 We use S3; we definitely don't use the web site. I've talked about potential improvements to the process here and here, but we're not there yet.
    – Aaron Bertrand Staff
    Commented Feb 8 at 13:12

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .