10

I kind of like flagging comments (as several mods will attest), and I have a few systems in place to aid me in flagging those comments (queues, etc.)

Now, it's recently come about that one of my more complicated attempts to dredge up obsolete comments on SO requires a copy of the entire comments table - like the one exposed on SEDE.

I tried abusing SEDE to get the comments, but it's smart enough not to fetch twenty-four million and let me download them as a .csv.

So how do I go about getting this data? (Ideally in a way that is as painless as possible for both SE and I)

I'm fine with just about any standard format, .csv or .sql being preferred. In the end, I want them in a MySQL DB.

Improve this question
7
  • 5
    I did a spit take when I read the title. Then I saw who wrote it and that you want to flag them all, and that makes perfect sense.
    – Bill the Lizard Mod
    Commented Nov 18, 2013 at 4:05
  • @Bill Lol, am I already getting a reputation in the mod queue?
    – Undo
    Commented Nov 18, 2013 at 4:06
  • What is your plan with this? If it ends up being some "let's clean up all the swearwords from comments" sting you're going to have to answer to me :)
    – Pekka
    Commented Nov 18, 2013 at 4:10
  • 3
    I don't know if the others have noticed yet, but I've been loving the obsolete flags lately.
    – Bill the Lizard Mod
    Commented Nov 18, 2013 at 4:10
  • 2
    @prob No, not that. It's an obsolete comment sting. I've been successfully whacking several hundred every day.
    – Undo
    Commented Nov 18, 2013 at 4:11
  • Sounds good, interested to see what you come up with.
    – Pekka
    Commented Nov 18, 2013 at 4:11
  • 1
    @BilltheLizard On Physics I've been using custom helpful messages to return the favor :P
    – Manishearth
    Commented Nov 18, 2013 at 4:14

2 Answers 2

Reset to default
4

You can download the StackExchange Data Dump from September 2013 from this site (for the full list, go here). It is available through a BitTorrent client.

There are other ways to grab the data, but I can't find them right now, and this is probably fastest and easiest anyway. Unfortunately, the format on this file is XML. Hopefully that won't be a problem.

Improve this answer
9
  • Any way to just get the comments portion? 14 GB is awful ginormous.
    – Undo
    Commented Nov 18, 2013 at 4:03
  • @Undo Not through the data dump, unfortunately. The data isn't dumped in segments.
    – user206222
    Commented Nov 18, 2013 at 4:03
  • Is this for SO only, or the whole network? Is it available for smaller sites, I.e. Physics?
    – Undo
    Commented Nov 18, 2013 at 4:05
  • Argh! Don't you think there are other things (than flagging all the comments as Bill said) to help physics? But in any case, see the "full list" that Emrakul linked...
    – dmckee --- ex-moderator kitten
    Commented Nov 18, 2013 at 4:07
  • @Undo This is for all StackExchange sites; however, when you select which files you're downloading in the torrent window, you can pick out which ones you'd like. Obviously, StackOverflow makes up the vast majority of this data.
    – user206222
    Commented Nov 18, 2013 at 4:08
  • @dmckee Lol, Physics is my testbed, primarily because it's smaller and I can't contribute anything decent there anyway.
    – Undo
    Commented Nov 18, 2013 at 4:15
  • @Undo Ah. I think we are developing a love/hate relationship with your flags. They are pretty consistently good, but...there are only four of us and a lot of that stuff is old.
    – dmckee --- ex-moderator kitten
    Commented Nov 18, 2013 at 4:22
  • @dmck Mind pinging me in Phys chat, so we can get this out of the comments?
    – Undo
    Commented Nov 18, 2013 at 4:23
  • 1
    @Undo Speaking of obsolete comments... ;]
    – user206222
    Commented Nov 18, 2013 at 4:24
2

You can use the API.

According to this quick query there are 24,527,873 comments on Stack Overflow.

The API will only return a maximum of 100 per query, and you will have to loop through until there are none left, meaning about 245k API calls. Current API limitations without a key are for 300 queries I believe, so you should go and apply for a key on stackapps so as to not incur the wrath of the powers that be.

You may want to start off with a more manageable comment count like on Physics where you only have 137,648 comments so far.

Here is a sample API call to get you a bunch of info on comments: click me for a wall of text

Here is the page to configure what information it returns: click me for a semi-manageable interface to manage the wall of text

(You can see it gets messy with a lot of info selected, so filter for just what you need)

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .