Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 157 Stack Exchange communities.

What is meta?
Here's how it works:
  1. Any Stack Exchange user can ask a question
  2. The community provides support, votes on ideas, and reports bugs
  3. Your voice helps shape the way Stack Exchange operates

Amazon provides Public Data Sets on Amazon Web Services (AWS):

Public Data Sets on AWS provides a centralized repository of public data sets that can be seamlessly integrated into AWS cloud-based applications. AWS is hosting the public data sets at no charge for the community, and like all AWS services, users pay only for the compute and storage they use for their own applications.

I believe this would be a good way to provide the Stack Exchange data dump in an easily accessible form. My questions are:

  • Does anybody here have experience with preparing and submitting a data set to Amazon for publication?
  • Does Amazon support publishers specifying their own license (such as the cc-wiki used here)? [I suspect the answer is yes, since there's a lot of Wikipedia data on there under the same license.]
  • How well does Amazon support a public data set that has updates every few months? What's the procedure for providing an update? Does the new data set replace the old one, or are multiple snapshots kept?

I'm willing to put in the work to make the data set available on AWS every time a new one is released. I asked the SE team about this and the response was basically "Interesting idea, ask on meta", so I'm now soliciting input.

My own situation is that I create a few things every time the data dump is released:

The data dump is getting fairly large, and takes a while to download (to this remote Pacific island with bandwidth limits and monthly data caps). I also might soon need more compute horsepower than what I've got available with my own hardware, since it now takes a few days to run through all the processing. I'd like to use EC2 to do the actual data crunching. I could of course just transfer it myself, but making it widely available seems like a good idea.

share|improve this question
Another question to ask is "how many users would actually take advantage of this if it's set up" (i.e. whether setting this up would be worth the effort). – Adam Lear Dec 27 '11 at 3:24
I would use it, and I don't think it's much more effort than just making it available to myself. :) – Greg Hewgill Dec 27 '11 at 4:50
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. Tanenbaum, Andrew S. (1996). That would be DVD or Blu-ray discs these days. – Peter Mortensen Jan 5 '13 at 15:46
Is there any chance that ebooks generator would be publicly available? BIG THANKS for books and I would like to see them updated with latest dumps! – Nemanja Boric Dec 28 '13 at 23:45
@NemanjaBoric: Please see – Greg Hewgill Dec 31 '13 at 7:02
Greg, thank you! Happy New Year! – Nemanja Boric Dec 31 '13 at 10:34
up vote 26 down vote accepted

We have applied to be a part of the Public Data Sets at Amazon. We'll keep you updated as we learn more about how this process works.

share|improve this answer
Wow, awesome! Let me know if I can be of any assistance. – Greg Hewgill Apr 2 '12 at 18:46
Any news on this yet? – Jeremy Banks Feb 3 '13 at 6:46
Somehow, I feel your promise seems to be not keeped... – Danubian Sailor Jan 14 '14 at 7:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .