14

For those curious about the progress of the quarterly data dump (and the automation behind it), has the current process started? Are we following the schedule described here?

Improve this question
5
  • 2
    For a moment I thought you're actually asking this, then relaxed when seeing the answer. Wonder if there's a way to indicate "This question has a self answer, don't worry" before the scared reader has to scroll down? :-)
    – Shadow Wizard
    Commented Oct 5, 2022 at 6:59
  • 2
    @ShadowTheKidWizard Sorry about that, I was just trying to pre-empt the inevitable. :-) Probably could have made it more like an announcement, but it felt more like something where I’d be updating the answer multiple times, as I have…
    – Aaron Bertrand Staff
    Commented Oct 5, 2022 at 11:04
  • 1
    Nothing to be sorry about, you couldn't make it better! :)
    – Shadow Wizard
    Commented Oct 5, 2022 at 11:14
  • 1
    @AaronBertrand: It might help once you accept your own answer; then it should show the checkmark icon next to the number of answers on the left side in the Questions list, I think.
    – V2Blast Staff
    Commented Oct 6, 2022 at 4:14
  • 1
    @V2Blast yeah I tried earlier today but it was before I hit the threshold and I forgot to go back
    – Aaron Bertrand Staff
    Commented Oct 6, 2022 at 6:04

1 Answer 1

Reset to default
19

The process completed as of Oct 5 at 05:28 UTC, however the files were corrupt and unusable. We’ve worked with Archive to determine the problem, resumed the upload, and this finished the last file some time yesterday.

This gets processed largely alphabetically. There are still cases where the receiving end slows down; We've made a few improvements to the resiliency and continue looking at ways to segment / parallelize.

Note that the process will often appear "stuck" when it gets to stackoverflow.com-*.7z due to the size of those files.

As a side note, we were a little ahead of the game this quarter thanks to some great work from one of our DBREs, Patrick Hurst, on improving both the automation on our side and the integration with archive's s3 API (which didn't exist - at least in its current form - back when we first moved from ClearBits or made minor improvements along the way).

Improve this answer
3
  • 2
    Looks like something didn't work very well, I am getting error "File is not an archive" for any file I try to decompress, using 7-zip software, also "View Contents" link is broken for all files. Can you please look into it?
    – Shadow Wizard
    Commented Oct 6, 2022 at 13:28
  • 1
    A report: Issues decompressing the October/2022 dump
    – This_is_NOT_a_forum
    Commented Oct 6, 2022 at 13:50
  • +1, was trying to figure out where to report and landed here. I have the previous quarter dumps which are working as expected.
    – vangap
    Commented Oct 6, 2022 at 13:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .