Since this was so well received, we have pushed a new data dump in early April and, for the foreseeable future, we plan to adhere to the schedule outlined in the question.
I've set a reminder to update this answer as each dump is delivered:
| Year/Q |
Started |
Data through |
Delivered (to all) |
| 2024 "Q0" |
March 3rd, 2024 |
February 29th, 2024 |
March 5th, 2024 ✅ |
| 2024 Q1 |
April 1st, 2024
12:28 AM UTC |
March 31st, 2024 |
April 2nd, 20241 ✅
8:32 AM UTC |
| 2024 Q2 |
August 2nd, 2024
(From observing file metadata) |
June 30th, 2024 |
August 14th, 2024 ✅
(Late due to process changes) |
| 2024 Q2 |
August 28th, 2024
(From observing file metadata) |
June 30th, 2024 |
August 29th, 2024 ✅
(Regenerated to fix issues) |
| 2024 Q3 |
October 3rd, 2024
(From observing file metadata) |
September 30th, 2024 |
October 7th, 2024 ✅ |
The first two deliveries were a lot earlier than you probably anticipated.
How'd we do that?
Well, in March, the process looked like this - we'd wait until Sunday, then process each Data Explorer database in sequence. Once Data Explorer was fully refreshed, we'd start the Data Dump process - extract each of the 8 tables to XML, then compress, one database at a time. Finally, we'd start the upload process, which would also upload one file at a time:
The diagram isn't to scale, but the times are accurate: it took over 70 hours to deliver the data dump for March once it started (March 3rd), and it required an additional Data Explorer refresh on a non-Sunday.
Since then...
...we have made several improvements to the process:
- Removed the dependency on Data Explorer, so the data dump can start shortly after midnight (UTC) on the first day of the quarter. It no longer waits for (or disrupts in any way) the Data Explorer processing; even if the first day of the quarter happens to also fall on a Sunday, they can now run concurrently.
- Made XML file creation and zipping happen in parallel, meaning uploading can start a lot sooner. We observed more than an 8X improvement here, and Data Explorer should see about a 4X improvement going forward.
- Eliminated ragged end times, so all data ends just before midnight (UTC) at the end of the previous period. This is true for weekly Data Explorer refreshes as well.2
- Made uploads happen in parallel. I was nervous about throttling and retries but, while we are still a little handcuffed by the upload speed to Internet Archive3, the speeds were consistent - even with multiple files uploading at the same time. The upload completed in just over 28 hours, cutting the last upload time almost in half.
All told, almost a 60% reduction in runtime, plus this is measured from the beginning of the quarter, instead of from the quarter's first Sunday at midnight UTC:
Without question, this was the fastest delivery in our history, and we have more improvements planned.
Minor differences you may have noticed before I did
This is a journey. There were a few minor differences observed by an anonymous edit that I can explain and, for the most part, have already fixed:
- Some data types were rendered incorrectly. For example, milliseconds were truncated from
datetime values that didn't have enough precision. Early on, I discovered an inconsistency with bit columns rendering as 0/1 instead of True/False, but this one slipped by me (mostly because most values do have enough precision).
→ status-completed
- Tags are delimited differently. As reported by @starball here, flattened lists of tags appear in the XML as
|this|format| when they should be an XML-safe variation of <this><format>. This will be corrected for the next data dump or maybe sooner.
→ status-completed
- Minor formatting differences. The previous version of the output would have two spaces before each
<row element, and a space before the ending />. In the new format, those spaces were missing. Similarly, 
 was rendering as 
.
→ status-completed
- XML files were encoded using UTF-16. I validated the content of files in the new process, but failed to notice they were encoded incorrectly and were larger as a result (at least pre-compression). While this may not affect everyone, it makes little sense to bloat the base files or disrupt the established encoding.
→ status-completed
Posts included deleted posts from PostsWithDeleted. This again had to do with validation - I spot-checked files for each table, but for sites small enough that I could manually open the files. Those sites I spot checked? Also happened to not have any deleted posts, so they looked exactly like files produced using the old method. (Deleted posts are pushed to SEDE, but not to the data dump, as described here.)
→ status-completed- XML files are no longer guaranteed to be sorted by
Id. This is true (though you might only see a difference on larger sites like Stack Overflow). The reason is that we enjoy incredible performance gains using parallelism, but we'd throw those gains away (and maybe more!) by forcing a re-sort after gathering streams. Presumably, people are loading these big files into their own systems, and not perusing multi-GB XML files in Notepad; those systems should support any subsequent sorting/indexing you need. (In truth, these weren't intentionally sorted by Id before; it was just a coincidence that the execution plan performed a single-threaded scan against the Id-leading clustered index.)
→ status-bydesign
These corrections marked status-completed have been applied, and a new batch of files has been successfully uploaded, with the last file finishing April 7th at 19:38 UTC.
Thank you for your continued patience as we try to further improve this deliverable.
1. Most files were actually available on Internet Archive a couple of hours into April 1st, but a handful of files take a lot longer. Subsequently, a new batch was uploaded with the fixes mentioned above, with most files arriving on April 6th, and the remainder on April 7th. The changes to account for these corrections will - at least initially - result in a longer "dump time" (the 51 minutes in the diagram above). However, since upload speed is our largest bottleneck by orders of magnitude, I plan to offset that bump by also spending a little more time up front on more aggressive compression.
2. This didn't make either process faster; it just feels a whole lot cleaner.
3. We get about 0.25mb/s, most of the time, with occasional bursts of up to 0.6mb/s. Our largest file, stackoverflow.com-PostHistory.7z (369 GB / 39.2 GB compressed), took over 28 hours to upload. While splitting that file up into chunks would allow us to deliver it faster, that would probably break folks, so we'll keep hoping IA will work with us on more efficient file transfer instead. In the meantime, higher compression should help a little - though reverting to UTF-8 did not help much at all, as the difference was largely compressed away anyway.
