13

Since the Stack Overflow data dump will occur monthy, it makes sense to be able to be notified about a new dump and download it. The current solution has been to post a link to the .torrent file on the blog, but this process requires manual intervention on the part of the producer and consumer of the data.

A better solution would be to create an RSS feed that would automatically be updated after the end of every month. Many BitTorrent clients have the ability to automatically download from an RSS feed, which would be convenient for obtaining the latest dump automatically. I imagine a situation where as soon as a month is over a script runs on Stack Overflow to dump the data, compress it, create the .torrent, update the RSS feed. Within minutes users/servers across the world automatically see that there's an update and proceed to download, effortlessly creating a worldwide content distribution network.

To get started, here's a site that describes how to create a valid BitTorrent RSS feed.

Improve this question
3
  • Good idea, but I also see the other side of it where everyone jumping on the new torrent would hammer the server quite a bit, especially before others start seeding.
    – Ian Elliott
    Jul 1, 2009 at 22:16
  • There are techniques to deal with that Like super seeding (en.wikipedia.org/wiki/Super-seeding). Overall, I think an automated process is called for here.
    – Kyle Cronin
    Jul 1, 2009 at 22:38
  • I still agree that an automated process is helpful, but I'm understanding if its not. There's also no way to force super-seeding.
    – Ian Elliott
    Jul 1, 2009 at 23:17

1 Answer 1

Reset to default
1

We'll just have a category in WordPress and you can subscribe to that category post feed if you want to be notified when new data dumps are available.

https://blog.stackoverflow.com/category/cc-wiki-dump/

RSS:

https://blog.stackoverflow.com/category/cc-wiki-dump/feed/

also note that we are now hosted on Legal Torrents:

http://www.legaltorrents.com/creators/146-stack-overflow-data-dump

They might have RSS feeds as well!

Improve this answer
1
  • It will probably be possible to make a 'real' BT RSS feed from this feed. It won't be automated on your end, but at least I can automate mine.
    – Kyle Cronin
    Jul 2, 2009 at 3:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .