12

GitHub announced in mid-November 2019 about their Code Vault Program. They plan to capture a snapshot of every active public repository on 02/02/2020 and preserve that data in the Arctic Code Vault.

Are there any plans similar to this for the Stack Exchange Network?

To be clear : Is there any possible future possibility where one can find the Stack Exchange Network deep within the Permafrost layer?

Improve this question
8
  • 3
    Why? Doesn’t the data dump contain revision history so that you can see what each post looked like over time?
    – Laurel
    Dec 7, 2019 at 14:12
  • 1
    Similar questions: meta.stackexchange.com/q/331125/282094 and meta.stackexchange.com/q/306593/282094.
    – Rob
    Dec 7, 2019 at 16:23
  • 2
    @Laurel The data dump is missing images as well as deleted questions/answers/comments.
    – Franck Dernoncourt
    Dec 7, 2019 at 19:06
  • One could probably add images to the datadump. That would come close. Problem would be that's not clear if the copyright of the images allows that. Or is this a clear cut case?
    – NoDataDumpNoContribution
    Dec 7, 2019 at 20:58
  • 1
    Is GitHub actually allowed to do that?
    – NoDataDumpNoContribution
    Dec 7, 2019 at 21:00
  • 6
    Create a GitHub repo containing the Stack Exchange data dumps or a mirror pulled from the wayback machine. Problem solved 8-)
    – Marco13
    Dec 7, 2019 at 21:43
  • @Marco13 Do you know how long it takes to get millions of pages from the wayback machine? Anyway you would have to delete all SO graphics and icons. As by the TOS, the design of this is property of the company. Not to speak of the copyright of the included graphics.
    – NoDataDumpNoContribution
    Dec 8, 2019 at 8:23
  • @Trilarion That wasn't entirely serious. The data dumps could actually be uploaded, but in the end, it doesn't really matter.
    – Marco13
    Dec 8, 2019 at 14:41

1 Answer 1

Reset to default
3

Is there any possibility where one can find the Stack Exchange Network deep within the Permafrost layer?

Not sure if the Internet Archive is deep and cold enough for you, but the company does quarterly data dumps at: https://archive.org/details/stackexchange

They include all content except deleted content and included content from third party sites (images, online regex testers, JavaScript testers, etc.). The layout is not included. This may come close to what you asked for and SE does it already for many years.

But if you want more, there is an organization called the Archive Team that has dedicated itself to doing such things. They might start archiving the whole of the Stack Exchange Network including images and other stuff for good if such a project were proposed. A quick search on their site returns no active project in this regard though.

And there are obstacles too (as there are for Github, I doubt they archive private repositories for example). The site design here is property of the company and one would need permission from them before archiving. Even worse, the linked and included external content is probably all sorts of mixed copyright. I'm not a lawyer, it may all be harmless, but I can also foresee legal problems there.

Improve this answer
4
  • 2
    Wouldn't be the first time the company has just completely ignored the complications of [mixed] licensing...
    – Lightness Races in Orbit
    Dec 14, 2019 at 22:53
  • 1
    Note that one comment states «I downloaded the Stack Overflow posts dump ('stackoverflow.com-Posts.7z') on 13 Sept 2019 which is lastly modified on 4 Sept 2019 according to archive.org. However, I found the latest post in the dump with date 11-03-2018 which does not make sense to me. I am not sure what the modified date of the dump means?» || Do we know whether the dump is complete?
    – Nemo
    Jan 18, 2020 at 13:08
  • @Nemo There was a thread about it. Something like this was observed before and was rectified by the company. Will try to find it. I hope it's complete. Don't want to download the GBs from IA, which is relatively slow, for nothing.
    – NoDataDumpNoContribution
    Jan 18, 2020 at 16:53
  • 1
    @Nemo I found the issue here Stack Overflow Outdated Dump and there it says it should have been fixed. Also there is a new data dump from 2019-12-02 currently available. I have locally only an older one from 2019-07-02 but that one seems to be okay. Also note, that for principal reasons it's difficult to determine whether a given dump is really complete. You can only really look at dates / number of lines in a statistical manner.
    – NoDataDumpNoContribution
    Jan 20, 2020 at 15:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .