18

I'm looking into running a machine-learning project on EL&U to detect low-quality posts. To do that, I need a large dataset - and to get that, I need to get the text from a large number of deleted questions.

  • I can't use the API; deleted posts aren't included in it.
  • I can't use SEDE: the PostsWithDeleted table lists NULL for the Body field of deleted posts.
  • I can't script it: not only do I have insufficient reputation on EL&U, but I also managed to get myself rate-limit-blocked while testing a script on Hardware Recs.

So, how else can I get this data?

Improve this question
5
  • 1
    Even if you had 10k, you'd still only be able to search for your own deleted posts. Without the URLs, I fear you need a mod-assist.
    – Monica Cellio
    Commented Dec 13, 2016 at 2:51
  • I probably do, @Monica, but I don't know what a mod could do either. I got myself blocked on one of my diamond sites trying to do this, so there still needs to be something else for a mod to do.
    – ArtOfCode
    Commented Dec 13, 2016 at 3:03
  • A mod can search for all deleted posts (or a narrower search including deletion). But that's through the UI, not scriptable, so the mod would still have to collect the URLs for you (or your hypothetical 10k user), who would then have to visit them one at a time to collect the text. In other words, it would be a manual process. It's a pity there's no way to get this through an API or SEDE.
    – Monica Cellio
    Commented Dec 13, 2016 at 3:09
  • @MonicaCellio that's exactly the thing I tried to script. It would work, but the rate limit is set such that collecting all the data would take a long time.
    – ArtOfCode
    Commented Dec 13, 2016 at 3:11
  • @ArtOfCode it's a little late, but meanwhile I do have enough reputation to download all deleted EL&U posts. You probably have shifted your attention elsewhere, but if you want to revive it, you know where to find me.
    – Glorfindel Mod
    Commented Jun 13, 2020 at 19:09

1 Answer 1

Reset to default
2

I'm a little late to the party, but if you have a computer running macOS, and sufficient reputation on the site, you can use the Revision Downloader AppleScript that I wrote. It needs just a list of post IDs (you can find those in SEDE in the PostsWithDeleted table) as an input file, and will download the source Markdown for all revisions of those posts in separate text files. I used it to get a collection of spam and rude/abusive posts but it could work for your scenario as well.

Improve this answer
5
  • Cool script, but ... not only do I have insufficient reputation on EL&U ...
    – rene Mod
    Commented Feb 1, 2018 at 16:54
  • I know, and neither do I (so I can't help Art) but it might help other people with the same problem.
    – Glorfindel Mod
    Commented Feb 1, 2018 at 16:57
  • Okay, fair enough. Just wanted to make sure you noticed it (I didn't and was happily wondering why I got 404's when I tried what your script does ...)
    – rene Mod
    Commented Feb 1, 2018 at 16:59
  • link dead? why is it gone? how did it work if it didn't use the API or SEDE? just scraping with the user's credentials?
    – starball
    Commented Aug 31, 2023 at 21:26
  • @starball it was deleted by Community because nobody found it useful :(
    – Glorfindel Mod
    Commented Sep 1, 2023 at 7:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .