7

I've been trying to implement a Machine-Learning based version of the "Smoke Detector" system by Charcoal, which uses human-crafted regular expressions to currently detect spam. It's already a great and highly accurate system, but using Machine Learning may reduce the number of False Negatives & Positives.

The great team at Charcoal was kind enough to provide a Data Dump of all the spam Charcoal has collected. However, this doesn't seem to be enough data for my deep learning systems. I'm trying to train an extremely deep learning model which combines CNNs, GRUs, and parallel models.

While I'm getting reasonably good accuracy, I need more data. Is there a way I can access deleted spam posts from the Stack Exchange network (not just SO in specific)? Of course, all to improve the quality of the content on the SE network :)

Also, this post isn't a duplicate of Obtaining text of spam/offensive posts (or any other related posts) because I'm not asking for all deleted posts or spam posts in the data dump, I'm asking for a dataset for myself to work with in Machine Learning tasks.

In fact, I've got a Keynote at the Watson Summit in New Zealand in a few days, and I plan to talk about this technology there as well: https://www.ibm.com/nz/watsonsummit/ (I'm Tanmay Bakshi)

NOTE: I'm currently working with 60,000 rows of data, but I'm thinking 100,000 (or as many as possible) would be much better to work with.

5
  • I've edited the post to show why I believe it's not a duplicate :)
    – TajyMany
    Commented Aug 9, 2017 at 9:11
  • "I'm not asking for all deleted posts or spam posts in the data dump, I'm asking for a dataset for myself to work" I don't see the distinction... how is what you're asking for any different?
    – Cai
    Commented Aug 9, 2017 at 9:23
  • 3
    Go to the contact page - explain what you are doing, what you want and perhaps you can come to some sort of agreement with Stack Exchange (possibly some kind of NDA on the contents of the actual posts - to allow you to get them all).
    – Oded StaffMod
    Commented Aug 9, 2017 at 9:25
  • 1
    Thanks @Oded! I've reached out, and I hope we can come to an agreement :)
    – TajyMany
    Commented Aug 9, 2017 at 18:28
  • @TajyMany Did anything come of this?
    – Blue
    Commented Nov 8, 2018 at 20:18

0

Browse other questions tagged .