25

We are in the process of automating the community data dump process and have reached the stage where we need to decide how and where to host the exported files.

We expect to have one data dump for all graduated/trilogy/MSO on a monthly basis, just under 12GB for the 7zipped archives at this time.

This is your data, so we're asking: Where would you like it hosted? Where is it easiest for everyone to access?

Options we had considered:

  • Self hosting. We can do this, but would like control of these files to remain with the community and outside of Stack Exchange hands, in case we go away.
  • ClearBits. We have used them before, but have found issues with upload stability.
  • Amazon S3. We have not been able to get a reliable contact there.
  • BitTorrent. The question here is again one of hosting the tracker, if this is the preferred option.

What are your preferences? Do you have better options?

Update:

Due to popular demand, the March 2013 data dump has been uploaded to clearbits - you can find it at:

http://www.clearbits.net/creators/146-stack-exchange-data-dump

We wanted the data dump to be available as soon as possible and we will keep discussing the longer term solution with the community.

Improve this question
18
  • 5
    One thought that came to mind was leaving a copy with Archive.org? Not necessarily for them to be the main download provider - more for archiving purposes. I have no idea whether they'd even be interested, though.
    – Pekka
    Mar 6, 2013 at 11:47
  • 1
    @Pekka웃 - I thought so too, but we would need a contact there, as their current options for hosting are Create and upload a new movie, audio recording, live concert recording, or book.. As the data dump doesn't fall under these, we need to ask someone.
    – Oded
    Mar 6, 2013 at 11:53
  • I think Jeff had some contacts there, it might be worth asking him.
    – Pekka
    Mar 6, 2013 at 11:54
  • 1
    @Pekka웃 - certainly something to look at, though I wouldn't want archive.org to be the primary source of the data dump. Great for archiving, as you said.
    – Oded
    Mar 6, 2013 at 12:29
  • 2
    yeah. It would be cool for them to have copies because they're the only non-profit entity that I can think of that actually has a mission of preserving data for future generations. Re the actual day-to-day distribution, maybe ask Google whether they want to sponsor or co-sponsor a download location? They're likely to stick around, they are experts in high-performance data delivery, and already do similar things with their JavaScript CDN
    – Pekka
    Mar 6, 2013 at 12:36
  • Why don't we just keep doing what we are already doing?
    – Doorknob
    Mar 6, 2013 at 13:12
  • 2
    @Doorknob - The process is not currently automated. We have various pain points with ClearBits (which is what we have been using). We are also looking for what the community wants - not just what we currently do.
    – Oded
    Mar 6, 2013 at 13:22
  • Oh. Okay, that makes sense.
    – Doorknob
    Mar 6, 2013 at 13:24
  • The problem with any service is that if it should be there forever it needs to be stable, permanent and probably free (SE have to pay otherwise). Why not combine the two "solutions". Archive it somewhere, like archive.org and then self-host as the primary source.
    – ben is uǝq backwards
    Mar 6, 2013 at 13:38
  • 7
    If possible, my preference would be for an S3-backed torrent. As long as you keep supporting it, this torrent will have S3 as a fast seed so that everybody can download it. But even if you stop supporting it, the torrent will still work as long as anybody else is still seeding the data (i.e. it's not a private torrent that will stop working without the tracker). It is possible to figure out the HTTP download URL by looking at the torrent data, but most users probably won't do that.
    – Jeremy
    Mar 6, 2013 at 14:17
  • 2
    @JeremyBanksᐛ Convert that to an answer!
    – Brian Webster
    Mar 6, 2013 at 22:30
  • @GeorgeWBush Done, and then updated now that I realize the feature might not work properly.
    – Jeremy
    Mar 7, 2013 at 0:05
  • 1
    @AshRj - We are working on it. We have had a hitch with ClearBits (upload limit appears to be 10GB) and we are working on it. The torrent will have a separate file for each site.
    – Oded
    Mar 10, 2013 at 14:52
  • 2
    @roblev - It is up an ready. Get the March 2013 data dump at: clearbits.net/creators/146-stack-exchange-data-dump
    – Oded
    Mar 18, 2013 at 9:05
  • 1
    Quick note: The clearbits seeder seems rather slow. Seems to be capped at around 256Kbyte/s. With only being able to connect to 2 other seeders, I'm maxing out at 1.5MByte/s (despite my actual speed being 100MB down/up). Maybe you could host it with clearbits AND seed it with a stackexchange server so it's not unbearably slow?
    – Earlz
    Jun 3, 2013 at 16:49

4 Answers 4

Reset to default
10

To suggest one combination from the comments -

How about asking archive.org whether they want to collect SE data dumps on behalf of the community for safekeeping? They're an established entity, and archiving knowledge is their mission.

With archive.org having a copy, the actual day-to-day distribution of the data dump to interested users could easily be done by Stack Exchange itself. In case SE goes away, nothing is lost because archive.org has a copy; and how / where to make that available for large-scale distribution then... that's a different question for a different day (which may never come!)

Improve this answer
1
6

I would prefer a torrent backed by Amazon S3.

A torrent can be generated for any file that is publicly accessible on S3 by appending ?torrent to its URL. Stack Exchange Inc. could then redistribute the torrent file without telling people the original S3 URL so that they would actually use the torrent and not just grab the file with HTTP. (It's possible to figure out the download URL from data in the torrent file, but hopefully the majority of users wouldn't.)

The torrent will be seeded as long as SEI makes it available on S3. However if SEI or Amazon stops supporting this, the torrent can still continue without them; because it's not marked as private users can continue to download from other peers without support from Amazon's tracker.

The major drawback of this approach is that it can only be used to generate a torrent for a single file, not a collection of files. The dump would need to be consolidated into a single file or split across multiple torrents. One possibility going forward would be for torrent of new dumps to only contain data that has been added or changed since the last dump. This would keep their individual sizes relatively small and allow users to update their local copies without redownloading the data they already have.

I really like this idea in theory, but I don't really know how well the feature actually works. I've tried it out a few times (here's an S3 ?torrent download for you to try). My unscientific impression has been that Amazon's seed is activated after the ?torrent file is first downloaded and shuts down after the torrent is inactive for some amount of time. It is never restarted, even if the ?torrent file is downloaded from Amazon again. Later downloaders seem to be out of luck.

Maybe there's some way to avoid this, but the only official docs I could find don't mention it and haven't been updated in seven years... we probably shouldn't rely on this feature.

As sth remarked, why can't Stack Exchange Inc. just seed a torrent? It would be supported by you but wouldn't depend on you. This doesn't seem like it would be too demanding on your servers or network.

I also don't understand why you would need a contact at Amazon for S3 to be viable. They have bulk import options available already, and I don't think this would be enough traffic that it would justify custom pricing or anything (unless due to some sort of promotional thing). Is there another reason?

Improve this answer
9
  • 1
    Couldn't they just as well seed the torrent from a Stack Exchange server?
    – sth
    Mar 6, 2013 at 23:38
  • @sth Yeah, I'm not sure why that wasn't considered in the question. Updated my post.
    – Jeremy
    Mar 7, 2013 at 0:18
  • About the traffic I can say that I have a server seeding the data dumps since they were released, and in total the traffic used by that until now is about 1.6 TB. You won't get any special price from Amazon for that.
    – sth
    Mar 7, 2013 at 1:44
  • @JeremyBanks While I was having a nap, the seed woke up and sent me the latest file that it hadn't been seeding earlier. Heh.
    – Jeremy
    Mar 7, 2013 at 2:40
  • @sth - It was considered. We wanted community input before any decision was made in regards to a permanent solution.
    – Oded
    Mar 7, 2013 at 9:44
  • After looking further - the standard S3 bittorrent offering will not work - torrents are limited to under 5GB and only work on single files. This means that the 9GB+ SO archive cannot be directly used - I think people would also like the choice of what archives to download (a single file with everything doesn't sound right to me).
    – Oded
    Mar 7, 2013 at 10:13
  • Along with what @Oded said, I would like things to be split up more. In the more recent data dumps, they added "Post History" which is, I think, the biggest part of the download, and wasn't even something that I wanted, but because it was in the same 7zip file, I had to download it anyway.
    – Kibbee
    Mar 7, 2013 at 14:14
  • @Kibbee I agree. I favour splitting it up as much as possible. Relatedly, it's unfortunate that the dumps are in XML, since that inflates the size so much. If a more compact format were used, it might justify non-compressed dumps so that one could more easily download only the information they want. (I mean, you could just split up the tables into multiple 7z files instead... but I'd still prefer uncompressed files because it mightn't be too hard to just grab new records from the end of a table without downloading the entire thing.)
    – Jeremy
    Mar 7, 2013 at 15:07
  • Personally, I think it would be nice of we could download diffs every day/week/month so as to cut down on the amount of data we needed to download. Issue a full download every month or so, but then have weekly diffs that can be applied so we don't have to redownload the same data every month. 12 GB is a significant part of my download cap.
    – Kibbee
    Mar 7, 2013 at 15:48
1

Maybe the hosting of Stack Overflow / Stack Exchange data dumps would be of interest for some universities to host. For instance at osuosl.org, but I am fairly sure, that others will be interested to include data dumps in their mirrors too, if asked. This may provide their students and researchers, with the opportunity to work more efficiently with data worth researching.

Companies come in and out, but universities usually stay.

Other options to try: - Contact mirrors that already host open source projects (as those hsting multiple Linux distros, apps and such). - Contact sourceforge, google code and other providers of similar services, which are open source friendly by design. - Contact Tier 1 and other major backbone ISPs - they may need that kind of traffic and publicity. - Contact Internet Exchange Points - just in case.

All of those probably will support hosting of data dumps in a long term, even if something goes wrong with the live sites, but please, don't go away!

Improve this answer
3
  • It's already hosted on Clear Bits. (This discussion was ended with that result)
    – hichris123
    Jan 4, 2014 at 19:01
  • At my end Clear Bits seems to be offline. It seems that they are offline for a long time. That's why I stumbled at this topic.
    – WebTech
    Jan 5, 2014 at 11:14
  • Sure. The dumps are very small so there should be no problem, except that such mirrors usually like to have an rsync target to pull from. See meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps for some examples.
    – Nemo
    Jan 24, 2020 at 20:33
0

There should be a more recent backup on Amazon S3 (creating a torrent would be a plus, as it was mentioned previously), but older backups (older than one month) could be moved to Amazon Glacier. Amazon Glacier is currently one of the best solutions for cheap, long term, secure backup.

Improve this answer
2
  • 3
    Amazon Glacier seems neat as an internal archival/backup service, but as far as understand it's not possible to make content stored in Glacier publicly accessible.
    – Jeremy
    Mar 6, 2013 at 23:17
  • Thanks for making me aware of Glacier - that's where my work backups will go from now on! And it's dirt cheap... cool.
    – Pekka
    Mar 7, 2013 at 12:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .