18

I'm used to finding the new dumps during the first week of a quarter, but now it's the 15th and I'm constantly refreshing https://archive.org/download/stackexchange.

(hopefully everyone is doing well)

Is this coming out soon?

Improve this question
9
  • Creating the data dumps and uploading to public platform isn't something Stack Exchange is obliged to do, it's just a bonus. I'm afraid if we'll nudge them too much about it, they'll just stop doing it. Heck, they almost closed the meta sites (including this one) not so long ago, just because the users annoyed few key employees.
    – Shadow Wizard
    Commented Mar 16, 2022 at 8:15
  • 5
    @ShadowWizardSaysNoMoreWar "I'm afraid if we'll nudge them too much about it, they'll just stop doing it." Really, they are that miffy? Well, for some users, the availability of the data dump might be more than a bonus but a requirement for continued participation. Maybe users can be capricious too.
    – NoDataDumpNoContribution
    Commented Mar 16, 2022 at 10:20
  • 2
    I would give them a few more days. It's not normal times and who knows, maybe this data dump isn't really automatized and needs manual steps and the person in charge might be on holiday or so. However, checking back after a reasonable time (say end of March) is reasonable. The company should be committed to these data dumps (they are more than just a bonus).
    – NoDataDumpNoContribution
    Commented Mar 16, 2022 at 10:23
  • @Trilarion it's not automated, DBA need to manually do stuff to both create it and upload to archive.org, as far as I know. And I'm sure they don't just "forget", hence why I disagree with nudging them, it just won't help.
    – Shadow Wizard
    Commented Mar 16, 2022 at 11:42
  • 1
    I might add that the employee responsible for uploading these dumps for the past few years recently left the company.
    – Sonic the Anonymous Hedgehog
    Commented Mar 16, 2022 at 21:29
  • Oh no, hopefully that doesn't mean we are not going to get more dumps out 🙏. Thanks @SonictheSaveUkraine-hog!
    – Felipe Hoffa
    Commented Mar 16, 2022 at 22:07
  • 1
    No, another staff member will take on the responsibility. I think they're still shuffling that.
    – Sonic the Anonymous Hedgehog
    Commented Mar 16, 2022 at 22:12
  • 1
    Oh, that's great to hear @SonictheSaveUkraine-hog — I got confused thinking that it might have meant the end. I love these dumps, and I'm eager to get the March data in.
    – Felipe Hoffa
    Commented Mar 16, 2022 at 22:21
  • 10
    Nothing wrong with gentle nudging, @shadow - the data dumps are important and moreover the processes & philosophies that underpin them are crucial to the long-term health of these sites! As for the intended shutdown of the meta sites... I've come to realize that whole fiasco was an instance of a common pattern for companies bringing on new people in preparation for sale - and ironically it likely would have gotten further if these sites had been quieter. Please let's never discourage productive questions out of fear!
    – Shog9
    Commented Mar 17, 2022 at 16:00

1 Answer 1

Reset to default
32

This is solved (more or less).

  • Our account has been restored.

  • Most data dumps are available (thanks Andy Mallon, aka AMtwo!), with the following exceptions:

    • Monero, which we skipped intentionally due to prior issues tripping malware scanners (see here and here)

    • Reverse Engineering, which seems to have tripped the malware scanner this time

    • Stack Overflow (which is large and takes the longest) was made available on March 21st:

      Screenshot showing SO data dump files

  • As a bonus, Jason Scott from the Internet Archive proactively reached out to us to make sure we will bypass the malware scanner in the future. We will attempt to provide the currently skipped sites in the next iteration of this free service we provide, once we have had ample time to test.

  • We do not maintain external copies of this data, such as BitTorrent. Brent Ozar talks about this (also free) service provided independently here: How to Download the Stack Overflow Database

  • If you are having issues, please comment below (rather than editing the answer).

Background & why we're late

The data dump is not automated, and requires a substantial amount of work.

This quarter, we've been focused on other higher priorities, including patching all of our SQL Servers and mitigating DDoS attacks (see here and here), but we'll get to the dumps soon - both manually providing the current ones and putting better automation in place so future ones are less of a burden.

Andy Mallon had already started on this quarter's work before this meta question was posted. Unfortunately, the Internet Archive has very sensitive malware triggers, and since post data is uploaded as is, it can be easy to trigger this (like we did today):

The item you have requested has a problem with one or more of the metadata files that describe it, which prevents us from displaying this page.

Items may be taken down for various reasons, including by decision of the uploader or due to a violation of our Terms of Use.

I can assure you we did not take it down or violate the terms of use; this happened because some content from a post on one of our sites (Reverse Engineering) tripped their filter – which automatically turns off all of our downloads.

Improve this answer
9
  • 14
    W00t! Didn't realize you were working for SE now - good on 'em!
    – Shog9
    Commented Mar 17, 2022 at 15:53
  • This has been a problem before. There is one network site that has at least one misbehaving post.
    – Luuklag
    Commented Mar 17, 2022 at 18:36
  • 2
    @Luuklag Yes, that is true! And we skip that one. But now it appears there is another. We are at the mercy of the Internet Archive's response time.
    – Aaron Bertrand Staff
    Commented Mar 17, 2022 at 19:07
  • @Luuklag If it's the same problem as September 2021, that was Monero.
    – Mast
    Commented Mar 17, 2022 at 19:55
  • 2
    So which post caused the malware scanner to trip?
    – Sonic the Anonymous Hedgehog
    Commented Mar 18, 2022 at 19:52
  • 1
    @SonictheSaveUkraine-hog We have no way to know which post it was because we don't upload posts individually. It's not really going to matter going forward anyway.
    – Aaron Bertrand Staff
    Commented Mar 18, 2022 at 19:53
  • @AaronBertrand Okay. Last time this happened, Internet Archive provided you with a link to VirusTotal so we could see which specific virus was being caught, which pointed us to the affected posts. Can that please be provided this time?
    – Sonic the Anonymous Hedgehog
    Commented Mar 18, 2022 at 19:58
  • 1
    @Sonic looks like this one did it
    – Aaron Bertrand Staff
    Commented Mar 18, 2022 at 20:16
  • 2
    Not sure why someone was editing instead of commenting, but the torrent reference pertained to the fact that all previous data dumps have been made available in that manner, either seeded via archive.org or (predating that) some other service (e.g. ClearBits). See: All Stack Exchange data dumps
    – Shog9
    Commented Mar 29, 2022 at 16:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .