It seems that the data dump is almost useless, just because all the data is really old. If people want to build applications, they are limited to the basic query's available in the API. But what if you want to do more?

Since uses for data are usually found after the fact, lets ignore that aspect for now. My question is why data dumps are limited to a monthly basis? Is it a technical limitation (computationally intensive, not enough bandwidth, not enough servers)? Is it an intensive manual operation for some reason? Is there a security issue? Or some other reason?

The preferred solution is a much more updated version. In the perfect world there would be a read only account with restrictions on what columns can't be viewed hooked up to the live db. Since that breaks basic sandbox rules, that's probably not possible. So why not a separate db that's synchronized with the live db on an hourly or 30 minute basis? Something that's a lot more live, better, and can be used as an alternative to the stack apps API for better statistical analysis.

  • 1
    Why the downvote? This is a perfectly interesting, legitimate question.
    – Graviton
    Commented Jul 31, 2010 at 6:40
  • 2
    @Ngu Downvotes on Meta tend to translate to "I don't want this".
    – Margaret
    Commented Jul 31, 2010 at 8:26

3 Answers 3

  • Because they're huge.
  • Because they take effort to make.
  • Because they are provided as torrents. 30 seeds to one monthly torrent is better than 30 daily torrents with 1 seed each.
  • * Because you don't need to do data-mining on a fine grained time scale. Or at least no one has proposed a convincing use case yet. Commented Jul 31, 2010 at 0:27
  • 1
    Step 1) Sync with "sandbox database". 2) Provide downloadable dumps as an HTTP download. 3) Enjoy
    – TheLQ
    Commented Jul 31, 2010 at 0:47
  • 6
    They're not a HTTP download. They're torrents. And they can't just distribute a plain DB dump - they have to remove information first (IP addresses, email addresses, passwords, etc)
    – Macha
    Commented Jul 31, 2010 at 0:49
  • Then make them an HTTP download. Cmon, were talking 825 MB here, not gigs upon gigs of files. And you can remove data before providing the link you know...
    – TheLQ
    Commented Jul 31, 2010 at 2:26
  • 1
    @Lord.Quackstar - That's also potentially a lot of bandwidth that SO isn't currently eating. Commented Jul 31, 2010 at 2:27
  • Cmon, at least look like your being creative. Outsource to other file hosters, provide read-only credentials to the "sandbox database" on <1 platform (this would actually make sense, limiting the amount of downloads people would need to do to import into their db of choice), or see if the used bandwidth would actually affect anything before blindly labeling it as huge.
    – TheLQ
    Commented Jul 31, 2010 at 3:11
  • @Macha, I don't know how much effort it would take, but why couldn't they do a monthly full dump and then include periodic diffs of the content. If they used rsync-able compression and then rsync for distribution instead of http they could do a full dump more often, but if use properly rsync would only download the bits that actually changed from the previous dump. It seems silly to be constantly redistributing the portions of the database that aren't changing.
    – Zoredache
    Commented Jul 31, 2010 at 6:49
  • @Lord «provide read-only credentials to the "sandbox database"»? Sounds like something I would know!
    – badp
    Commented Jul 31, 2010 at 17:41
  • @badp: Do you see database credentials? Do you see an actual mysql/oracle/auzre server url? No. All you have is the obscure Odata protocol. When I say database I mean database, not some stupid wrapper.
    – TheLQ
    Commented Aug 1, 2010 at 3:01
  • @Lord.Quackstar 885mb multiplied by however many users download it. Now it's gigs and gigs of stuff.
    – Macha
    Commented Aug 2, 2010 at 14:54
  • 1
    In comparison, the SO frontpage is 240 kb in size. So for each user who downloads the dump if they put it on their server, they could use that bandwidth to serve the SO home page ((885 * 1024 * 1024) / 240) times. Which is just under 4 million page views to use the same amount of bandwidth.
    – Macha
    Commented Aug 2, 2010 at 14:57

Since the entirety of the data is pretty large, it's not feasible to be releasing a full data set on a frequent basis. (It's an expensive database + I/O operation, as well as requiring a certain amount of site downtime so the data state is consistent.)

However, what could be feasible is having a full dump every 2 weeks or 1 month, and having incremental updates on a daily basis.

Most of the content in the system is essentially static*, so an incremental data set would be quite manageable. Simply include (a) new data items, (b) modified data items, and (c) indicators of deleted data items.

This could be an automated process that creates the update files and posts them to a repository of some kind. Based on the current size of the dump, these update files would be in the neighbourhood of ~1.4 MB each (for one day, across SOFU + Meta). (It may make sense to explicitly split the dump by site, but that's a separate discussion.) Since the content is cumulative, if full dumps are done every 2 weeks, for example, all that would have to be posted/active at once is one full dump plus up to 2 weeks' worth of daily updates for the time between now and the previous full dump.

Of course, there is still the issue of expensiveness of the operation and downtime.

* For another answer, I had run a query on Data Explorer to find the distribution of edited posts that are more than n days old, and I found that posts are overwhelmingly likely to be edited within a short period of time after being posted.

  • 1
    How expensive? How long is the downtime? Is it noticeable to the user?
    – TheLQ
    Commented Aug 1, 2010 at 7:45
  • @Lord: I can't really quantify the expensiveness, but exporting more than 5 GB of data (and only ever increasing in size) isn't going to be inexpensive. Yes, the downtime is noticeable. I think it's roughly 2-3 minutes for SO when you get a default "we're down for maintenance page."
    – Jon Seigel
    Commented Aug 1, 2010 at 13:22

Supposedly they were looking into making them more frequent, but possibly just for SEDE; I'm not sure if that's still under consideration or not

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .