What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 134 Stack Exchange communities.

Now that net neutrality is gone and Comcast and it's ilk are beginning to throttle BitTorrent, we should have a "legal" way to download the data dump.

I know there's third-party hosts, but they're often outdated and prone to 404ing. Why can't we have a plain ol' HTTP release?

share|improve this question
    
Are you going to pay for the bandwidth? We're talking 700mb per download, and rising. Not that I totally disagree... –  balpha Apr 28 '10 at 14:57
3  
How can a protocol be illegal? –  jmfsg Apr 28 '10 at 15:04
    
@Downvoter: Ask the US government. –  Phoshi Apr 28 '10 at 15:05
2  
@Downvoter: ask fibertel. They told me they were filtering it for "security reasons". To my surprise Telefónica has a great service. –  perbert Apr 28 '10 at 15:05
    
It's not how, it's what. Anyway, there are ways to circumvent technical barriers @voy, we are geeks after all... I use my full bandwidth with fibertel. –  jmfsg Apr 28 '10 at 15:38
    
@Downvoter: not when they don't send their technicians for 2 moths, leaving me without a connection because they had issued a wrong impedance connector, and the whole block had problems but I were the only one complaining, so they assumed it was a problem with viruses. On my Debian box. sigh I'll be happy if I never have to deal with their half-brained tech support. No sir, we don't support SSH connections. So you are running the Linux application? Have you tried uninstalling it and trying again? –  perbert Apr 28 '10 at 16:19
1  
@Downvoter: a protocol can be illegal if its use is prohibited by the law. In order to do that the protocol itself doesn't need to be evil; the fact that it can be used for malicious purposes is usually sufficient. See for example guns; how can guns be illegal? They don't harm anyone as long as you don't use them to shoot people. Same for bittorrent; it doesn't hurt anyone if used for legal purposes, but it has the potential to be misused, and thus is/will be/can be/might be illegal. –  Andreas Bonini Apr 28 '10 at 16:31
    
@Kop: are guns illegal? –  perbert Apr 29 '10 at 14:20
    
@voyager: in many countries, including mine, yes. –  Andreas Bonini Apr 29 '10 at 18:45

4 Answers 4

What isn't legal about http://clearbits.com and the torrent protocol?

There are usually http mirrors contributed by other users within a few hours of the torrent being posted.

As always see

http://blog.stackoverflow.com/category/cc-wiki-dump/

For the latest information on the data dumps.

share|improve this answer
3  
Comcast filtering bittorret had nothing to do about legality. –  perbert Apr 28 '10 at 15:05

I assume the barrier to this is the enormous amount of bandwidth needed to support such a download.

Instead of putting the load directly on one server, using BitTorrent, it's distributed over the community at large.

Yes, some ISPs are throttling BitTorrent, which is unfortunate because it's a completely legal method to download and share files. The whole issue surrounds the content of the files. Since ISPs would be sued into the stone age if they started filtering content, they chose to do something about the protocol instead, and that, of course, is a hot topic of debate.

That being said, even if an ISP is throttling your bandwidth to, say, 20 kb/s at all hours of the day, the data dump can be downloaded in less than half a day. For anyone wanting to download this stuff, it's not a big deal to leave your computer on overnight.

I would say that an official HTTP host of this content is unnecessary, although certainly not undesirable if it were available. And as Jeff mentions, users are free to host the files however they want after an original copy is obtained.

Alternate solution: PayPal me $5 and I'll send you a copy on a DVD. :)

share|improve this answer
3  
I had a ISP (Fibertel, something like the local Comcast) sending RST responses to both parties of any bittorrent connection within seconds of it being established. I got .2kbps for any given torrent. I can understand this guy's feelings, but the one at fault is his ISP. –  perbert Apr 28 '10 at 15:12
    
@voyager: Yeah, for sure. Every ISP does it differently, too, so it's kind of case-dependent. One of the ISPs here caps at 20-something kb/s in core usage hours. I think that for internet power users who would be downloading a data dump, it is less likely they would be on a throttled connection (unless there is literally zero choice), so I'm assuming this is a very small minority problem. –  Jon Seigel Apr 28 '10 at 15:33
1  
Copy on a dvd is not a bad idea. (not that i'm gona order one). Maybe stackoverflow should make it official –  Midhat Apr 29 '10 at 20:40

Look, I'm aware that in several places there are large ISP that don't give a rat's ass about their customers, and that filter everything non HTTP (and sometimes even that), but it's not SO's fault, it's your ISP's.

I live in a city that has at most 5 big providers, 3 of which only resell the other 2 pipes. The provider I used to have is like Comcast, only that with another name, After 6 months of taking their abuse, I just changed for a telcom ADSL provider and haven't looked back. They understand a bit better that they are supposed to be dumb pipes.

You wouldn't ask So to be available on Gopher only because your ISP is blocking HTTP, would you?

share|improve this answer

I have a little archive of data dumps here and usually also put new ones there rather fast.

But on the other hand the torrents are also well seeded and much more fail safe, so it really makes sense to download the data by bittorrent. That Comcast does stupid things doesn't mean that suddenly bittorrent as a distribution service is broken. It means that Comcast broke their own internet service, but this is all the problem of Comcast and their customers. If the customers don't like that they should complain to Comcast or change their service provider.

Thinking about it, maybe I should remove the data dump files from my server.

share|improve this answer
    
Wow. I've been looking for this old data for several days. ClearBits disappeared, the other HTTP mirror was shut down, and there are no peers on BitTorrent for several of them. I was just about to make a post asking if the data was lost forever when I came across theis post. I'm grateful that, in the end, you didn't decide to remove the data dump files from your server. Thanks! –  Jeremy Banks Mar 14 at 0:01
    
@JeremyBanks: I have added some more dumps that were on the server but not linked in that directory. So in case also need something from from 2010-09 to 2012-08 it's now downloadable. (There isn't a dump for every month, but I'm pretty sure it's everything that was released in that time frame). –  sth Mar 14 at 0:52
    
Beautiful, thanks again. I didn't even realize that the pre-ClearBits data dumps were released as torrents, and now I have those too. One funny thing: I notice that the for oldest dump, May 2009, you have a torrent for a .ZIP file, but the data in a .7Z file. If you remember: were the dumps released in both formats, and you just happened to download a non-matching pair? –  Jeremy Banks Mar 14 at 1:28
    
(Nevermind; the torrent used the 7Z file, but just had it mis-named as a ZIP file.) –  Jeremy Banks Mar 14 at 1:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .