10

I kind of like flagging comments (as several mods will attest), and I have a few systems in place to aid me in flagging those comments (queues, etc.)

Now, it's recently come about that one of my more complicated attempts to dredge up obsolete comments on SO requires a copy of the entire comments table - like the one exposed on SEDE.

I tried abusing SEDE to get the comments, but it's smart enough not to fetch twenty-four million and let me download them as a .csv.

So how do I go about getting this data? (Ideally in a way that is as painless as possible for both SE and I)

I'm fine with just about any standard format, .csv or .sql being preferred. In the end, I want them in a MySQL DB.

7
  • 5
    I did a spit take when I read the title. Then I saw who wrote it and that you want to flag them all, and that makes perfect sense. – Bill the Lizard Nov 18 '13 at 4:05
  • @Bill Lol, am I already getting a reputation in the mod queue? – Undo Nov 18 '13 at 4:06
  • What is your plan with this? If it ends up being some "let's clean up all the swearwords from comments" sting you're going to have to answer to me :) – Pekka Nov 18 '13 at 4:10
  • 3
    I don't know if the others have noticed yet, but I've been loving the obsolete flags lately. – Bill the Lizard Nov 18 '13 at 4:10
  • 2
    @prob No, not that. It's an obsolete comment sting. I've been successfully whacking several hundred every day. – Undo Nov 18 '13 at 4:11
  • Sounds good, interested to see what you come up with. – Pekka Nov 18 '13 at 4:11
  • 1
    @BilltheLizard On Physics I've been using custom helpful messages to return the favor :P – Manishearth Nov 18 '13 at 4:14
4

You can download the StackExchange Data Dump from September 2013 from this site (for the full list, go here). It is available through a BitTorrent client.

There are other ways to grab the data, but I can't find them right now, and this is probably fastest and easiest anyway. Unfortunately, the format on this file is XML. Hopefully that won't be a problem.

10
  • Any way to just get the comments portion? 14 GB is awful ginormous. – Undo Nov 18 '13 at 4:03
  • @Undo Not through the data dump, unfortunately. The data isn't dumped in segments. – user206222 Nov 18 '13 at 4:03
  • Is this for SO only, or the whole network? Is it available for smaller sites, I.e. Physics? – Undo Nov 18 '13 at 4:05
  • Argh! Don't you think there are other things (than flagging all the comments as Bill said) to help physics? But in any case, see the "full list" that Emrakul linked... – dmckee --- ex-moderator kitten Nov 18 '13 at 4:07
  • @Undo This is for all StackExchange sites; however, when you select which files you're downloading in the torrent window, you can pick out which ones you'd like. Obviously, StackOverflow makes up the vast majority of this data. – user206222 Nov 18 '13 at 4:08
  • Ah, okay. Thanks! – Undo Nov 18 '13 at 4:08
  • @dmckee Lol, Physics is my testbed, primarily because it's smaller and I can't contribute anything decent there anyway. – Undo Nov 18 '13 at 4:15
  • @Undo Ah. I think we are developing a love/hate relationship with your flags. They are pretty consistently good, but...there are only four of us and a lot of that stuff is old. – dmckee --- ex-moderator kitten Nov 18 '13 at 4:22
  • @dmck Mind pinging me in Phys chat, so we can get this out of the comments? – Undo Nov 18 '13 at 4:23
  • 1
    @Undo Speaking of obsolete comments... ;] – user206222 Nov 18 '13 at 4:24
2

You can use the API.

According to this quick query there are 24,527,873 comments on Stack Overflow.

The API will only return a maximum of 100 per query, and you will have to loop through until there are none left, meaning about 245k API calls. Current API limitations without a key are for 300 queries I believe, so you should go and apply for a key on stackapps so as to not incur the wrath of the powers that be.

You may want to start off with a more manageable comment count like on Physics where you only have 137,648 comments so far.

Here is a sample API call to get you a bunch of info on comments: click me for a wall of text

Here is the page to configure what information it returns: click me for a semi-manageable interface to manage the wall of text

(You can see it gets messy with a lot of info selected, so filter for just what you need)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .