26

A comment by shog9 piqued my interest on this.

FWIW, I'm happy to provide data on the accuracy of spam flaggers if requested... But please make it a separate question so I don't have to post that in a comment. Tease: it's probably a lot worse than you think it is.

Right now Charcoal has some anecdotal year-old data which suggests 95,4% accuracy across all sites.

I'd be interested in (specifically):

  • How accurate are spam flaggers in general?
  • Are there huge differences by site?
  • How many spam flags that end up being handled by a mod are declined?

Additionally (I don't know how easy it is to get that):

  • Are users more likely to also cast a spam flag on a non-spam post when one was previously cast on it?
  • How pronounced is that effect?
  • How many posts were actually wrongfully deleted as spam (That is, spam deleted and then undeleted later)?
7
  • You're also going to want to make sure to know if you are getting data that is on a per user-flag basis, or if it's on a per-post basis.
    – Makyen
    Commented Mar 8, 2018 at 10:05
  • 1
    "Are users more likely to also cast a spam flag on a non-spam post when one was previously cast on it?" Since there's no public indication that a post has been flagged, I'd assume that (without being notified elsewhere; e.g. by SD or a comment on the post) that would make little difference.
    – Cai
    Commented Mar 8, 2018 at 10:14
  • 6
    there are two statistics that are interesting, how accurate individual flags are, and how often posts were actually community-deleted by spam flags that shouldn't have been (= were undeleted later). Commented Mar 8, 2018 at 11:19
  • @MadScientist Added that to the list of questions.
    – Magisch
    Commented Mar 8, 2018 at 12:28
  • Jon's answer here put the declined flag rate on questions and answers at Stack Overflow at ~1.7%. That's for all flags, however. We'll also have to account for the "disputed" state on spam flags, which I'd lump into the inaccurate category. Commented Mar 8, 2018 at 16:12
  • @MadScientist The numbers for posts which were community-deleted by spam or rude/abusive flags and were undeleted should be split into those which were edited prior to being undeleted and those which weren't. Being edited prior to being undeleted could indicate that the spam flags were reasonable.
    – Makyen
    Commented Mar 8, 2018 at 23:05
  • 1
    @Magisch Are you looking for only spam flags or spam and rude/abusive combined, or those two types separately?
    – Makyen
    Commented Mar 8, 2018 at 23:06

1 Answer 1

12

Two bits of information are pretty easy to get here; the rest I'm going to skip for the time being as it's gonna take more time than I have to retrieve.

Helpful flags

This is pretty self-explanatory: if a flag was marked helpful, it counts; if it wasn't, it doesn't. Spam flags can be marked helpful in the following scenarios:

  • A moderator explicitly marks the flag helpful
  • The post accumulates enough spam flags to delete the post as spam (w/ locking, penalties, etc.)
  • The post is deleted for some other reason (author vote, privileged user vote, LQ review, moderator vote)

Unlike most other flags, spam flags can be retroactively disputed if a moderator decides it wasn't warranted; this can affect the outcome as well.

Here are some numbers for helpful spam flags, network-wide, for the past 365 days:

                Site                 SpamFlags HelpfulSpamFlags PctSpamFlagsHelpful
------------------------------------ --------- ---------------- -------------------
Stack Overflow                           36303            30448             83.87 %
Super User                               18245            18070             99.04 %
Server Fault                              1264             1232             97.47 %
Stack Apps                                 296              294             99.32 %
Seasoned Advice                            325              314             96.62 %
Home Improvement                           800              770             96.25 %
Game Development                           271              262             96.68 %
Arqade                                    1995             1948             97.64 %
GIS                                        199              189             94.97 %
Mathematics                                919              821             89.34 %
Photography                                317              298             94.01 %
Cross Validated                            173              166             95.95 %
Web Applications                          2340             2294             98.03 %
Webmasters                                 686              674             98.25 %
Ask Different                            21840            21789             99.77 %
Theoretical Computer Science                18               15             83.33 %
English Language & Usage                   901              878             97.45 %
Money                                     2243             2211             98.57 %
RPG                                        459              447             97.39 %
TeX - LaTeX                                376              369             98.14 %
Ask Ubuntu                               46473            46317             99.66 %
Unix & Linux                               524              503             95.99 %
WordPress Development                      863              828             95.94 %
Bicycles                                   297              273             91.92 %
Software Engineering                       391              380             97.19 %
Android Enthusiasts                       1837             1766             96.14 %
Board & Card Games                         148              140             94.59 %
Physics                                   1079             1035             95.92 %
Homebrewing                                 27               24             88.89 %
Information Security                      2588             2570              99.3 %
Writing                                    256              247             96.48 %
Electrical Engineering                     787              765              97.2 %
Graphic Design                           23860            23816             99.82 %
DBA                                        232              222             95.69 %
Science Fiction & Fantasy                  690              668             96.81 %
Area 51 Discussions                         22               21             95.45 %
Code Review                                271              262             96.68 %
Code Golf                                  543              527             97.05 %
Quantitative Finance                        84               71             84.52 %
Project Management                         107              107               100 %
Skeptics                                   343              339             98.83 %
Drupal Answers                           15880            15860             99.87 %
Physical Fitness                           347              328             94.52 %
Motor Vehicles                             401              393                98 %
Parenting                                  394              389             98.73 %
SharePoint                                 421              407             96.67 %
Music                                      302              296             98.01 %
Software Quality Assurance & Testing       246              240             97.56 %
Mi Yodeya                                   98               90             91.84 %
Astronomy                                 7330             7325             99.93 %
German Language                            149              138             92.62 %
Japanese Language                           75               75               100 %
Gardening & Landscaping                     95               91             95.79 %
Philosophy                                  29               28             96.55 %
Personal Productivity                      183              180             98.36 %
Travel                                    2136             2105             98.55 %
Cryptography                               131              127             96.95 %
Literature                                  26               25             96.15 %
Signal Processing                          165              164             99.39 %
French Language                             64               64               100 %
Christianity                                51               49             96.08 %
Bitcoin                                   1816             1774             97.69 %
Linguistics                                 22               21             95.45 %
Biblical Hermeneutics                       42               39             92.86 %
Economics                                   34               34               100 %
History                                    160              158             98.75 %
Bricks                                      41               41               100 %
Spanish Language                            42               41             97.62 %
Computational Science                       40               39              97.5 %
Movies & TV                                433              417              96.3 %
Chinese Language                           144              112             77.78 %
Biology                                    450              433             96.22 %
Poker                                      176              158             89.77 %
Mathematica                                 57               54             94.74 %
Psychology & Neuroscience                   71               71               100 %
The Great Outdoors                         164              160             97.56 %
Martial Arts                                35               34             97.14 %
Sports                                     108              106             98.15 %
Academia                                  1374             1345             97.89 %
Computer Science                            85               80             94.12 %
The Workplace                             4002             3977             99.38 %
Windows Phone                               50               46                92 %
Chemistry                                  272              250             91.91 %
Chess                                       53               49             92.45 %
Raspberry Pi                               128              121             94.53 %
Russian Language                            13               13               100 %
Islam                                      318              305             95.91 %
Salesforce                                 349              342             97.99 %
Ask Patents                                320              318             99.38 %
UX                                         489              479             97.96 %
Genealogy & Family History                  41               41               100 %
Robotics                                    28               27             96.43 %
ExpressionEngine® Answers                   75               73             97.33 %
Politics                                   106              106               100 %
Video Production                           122              115             94.26 %
Anime & Manga                              239              236             98.74 %
Magento                                    764              686             89.79 %
English Language Learners                  570              560             98.25 %
Sustainable Living                          35               33             94.29 %
Tridion                                      1                1               100 %
Reverse Engineering                         47               46             97.87 %
Arduino                                    983              975             99.19 %
Network Engineering                         65               62             95.38 %
Open Data                                   54               49             90.74 %
Freelancing                                 89               77             86.52 %
Blender                                     91               81             89.01 %
MathOverflow                              1635             1628             99.57 %
Space Exploration                           54               50             92.59 %
Sound Design                                71               68             95.77 %
Tor                                        109               96             88.07 %
Pets                                       172              163             94.77 %
Amateur Radio                               32               32               100 %
Italian Language                             5                5               100 %
Stack Overflow em Português                399              359             89.97 %
Aviation                                   510              504             98.82 %
Ebooks                                     137              137               100 %
Beer, Wine & Spirits                        60               58             96.67 %
Software Recommendations                   321              293             91.28 %
CS50                                         2                2               100 %
Expatriates                                231              230             99.57 %
Mathematics Educators                       12               11             91.67 %
Earth Science                               43               40             93.02 %
Joomla                                     191              186             97.38 %
Data Science                                74               73             98.65 %
Puzzling                                   325              316             97.23 %
Craft CMS                                   56               54             96.43 %
Buddhism                                    52               50             96.15 %
Hinduism                                   128              121             94.53 %
Community Building                         165              164             99.39 %
Startups                                   250              232              92.8 %
Worldbuilding                              650              626             96.31 %
スタック・オーバーフロー                       50               48                96 %
Emacs                                        3                2             66.67 %
History of Science and Mathematics          14               13             92.86 %
Lifehacks                                  180              168             93.33 %
Engineering                                 55               52             94.55 %
Coffee                                      50               39                78 %
Vi and Vim                                   3                1             33.33 %
Music Fans                                  57               55             96.49 %
Woodworking                                113              110             97.35 %
CiviCRM                                    150              136             90.67 %
Health                                     892              877             98.32 %
Stack Overflow на русском                 1135             1030             90.75 %
Русский язык                                57               51             89.47 %
Mythology & Folklore                        37               36              97.3 %
Law                                        161              160             99.38 %
Open Source                                 33               29             87.88 %
elementary OS                               14               14               100 %
Portuguese Language                          5                5               100 %
Computer Graphics                            8                8               100 %
Hardware Recommendations                    25               17                68 %
Stack Overflow en español                  327              258              78.9 %
3D Printing                                579              498             86.01 %
Ethereum                                   270              265             98.15 %
Latin Language                              12               11             91.67 %
Language Learning                            7                7               100 %
Retrocomputing                              20               20               100 %
Arts & Crafts                               72               72               100 %
Korean Language                              2                2               100 %
Monero                                      65               64             98.46 %
Artificial Intelligence                     86               83             96.51 %
Esperanto Language                           8                0                 0 %
Sitecore                                    16               14              87.5 %
Internet of Things                          49               48             97.96 %
Vegetarianism                               58               57             98.28 %
Ukrainian Language                           1                1               100 %
DevOps                                       4                3                75 %
Bioinformatics                               4                3                75 %
Computer Science Educators                  21               20             95.24 %
Interpersonal Skills                       698              689             98.71 %
Augur                                        5                5               100 %

Effective flags

Now, not every flag marked helpful was actually effective... As you might well guess from the description above; obviously a spam flag that's cleared by Low Quality review didn't end up doing much! You could argue that a spam flag which prompts a moderator - or even a post author - to delete a post is still effective in some sense... And you can go right ahead and argue that, but for the purpose of the next table I'm going to consider a spam flag to be helpful only if it resulted in the flagged post being deleted as spam - that is, either enough other people flagged it as spam or a moderator flagged it as spam - and then it stayed deleted afterwards.

Again, past 365 days:

                Site                 SpamFlags EffectiveSpamFlags PctSpamFlagsEffective
------------------------------------ --------- ------------------ ---------------------
Stack Overflow                           36304              22791               62.78 %
Super User                               18243              17541               96.15 %
Server Fault                              1264               1082                85.6 %
Stack Apps                                 296                282               95.27 %
Seasoned Advice                            325                249               76.62 %
Home Improvement                           800                724                90.5 %
Game Development                           271                207               76.38 %
Arqade                                    1995               1703               85.36 %
GIS                                        199                103               51.76 %
Mathematics                                919                557               60.61 %
Photography                                317                185               58.36 %
Cross Validated                            173                101               58.38 %
Web Applications                          2340               2196               93.85 %
Webmasters                                 686                299               43.59 %
Ask Different                            21840              21339               97.71 %
Theoretical Computer Science                18                  7               38.89 %
English Language & Usage                   901                821               91.12 %
Money                                     2243               1584               70.62 %
RPG                                        459                413               89.98 %
TeX - LaTeX                                376                243               64.63 %
Ask Ubuntu                               46473              45851               98.66 %
Unix & Linux                               524                368               70.23 %
WordPress Development                      863                576               66.74 %
Bicycles                                   297                170               57.24 %
Software Engineering                       391                286               73.15 %
Android Enthusiasts                       1837               1652               89.93 %
Board & Card Games                         148                108               72.97 %
Physics                                   1079                906               83.97 %
Homebrewing                                 27                 20               74.07 %
Information Security                      2588               2329               89.99 %
Writing                                    256                216               84.38 %
Electrical Engineering                     787                479               60.86 %
Graphic Design                           23860              23613               98.96 %
DBA                                        232                182               78.45 %
Science Fiction & Fantasy                  690                583               84.49 %
Area 51 Discussions                         22                 21               95.45 %
Code Review                                271                192               70.85 %
Code Golf                                  543                498               91.71 %
Quantitative Finance                        84                 28               33.33 %
Project Management                         107                 84                78.5 %
Skeptics                                   343                276               80.47 %
Drupal Answers                           15880              15380               96.85 %
Physical Fitness                           347                212                61.1 %
Motor Vehicles                             401                340               84.79 %
Parenting                                  394                302               76.65 %
SharePoint                                 421                244               57.96 %
Music                                      302                277               91.72 %
Software Quality Assurance & Testing       246                132               53.66 %
Mi Yodeya                                   98                 74               75.51 %
Astronomy                                 7330               7258               99.02 %
German Language                            149                132               88.59 %
Japanese Language                           75                 63                  84 %
Gardening & Landscaping                     95                 90               94.74 %
Philosophy                                  29                 14               48.28 %
Personal Productivity                      183                167               91.26 %
Travel                                    2136               1670               78.18 %
Cryptography                               131                103               78.63 %
Literature                                  26                 23               88.46 %
Signal Processing                          165                153               92.73 %
French Language                             64                 52               81.25 %
Christianity                                51                 39               76.47 %
Bitcoin                                   1816               1080               59.47 %
Linguistics                                 22                 12               54.55 %
Biblical Hermeneutics                       42                 21                  50 %
Economics                                   34                 18               52.94 %
History                                    160                 95               59.38 %
Bricks                                      41                 26               63.41 %
Spanish Language                            42                 22               52.38 %
Computational Science                       40                 25                62.5 %
Movies & TV                                433                379               87.53 %
Chinese Language                           144                 78               54.17 %
Biology                                    450                383               85.11 %
Poker                                      176                127               72.16 %
Mathematica                                 57                 31               54.39 %
Psychology & Neuroscience                   71                 49               69.01 %
The Great Outdoors                         164                129               78.66 %
Martial Arts                                35                 16               45.71 %
Sports                                     108                 75               69.44 %
Academia                                  1374               1164               84.72 %
Computer Science                            85                 69               81.18 %
The Workplace                             4002               3947               98.63 %
Windows Phone                               50                 35                  70 %
Chemistry                                  272                230               84.56 %
Chess                                       53                 38                71.7 %
Raspberry Pi                               128                 73               57.03 %
Russian Language                            13                  6               46.15 %
Islam                                      318                245               77.04 %
Salesforce                                 349                251               71.92 %
Ask Patents                                320                315               98.44 %
UX                                         489                431               88.14 %
Genealogy & Family History                  41                 36                87.8 %
Robotics                                    28                 10               35.71 %
ExpressionEngine® Answers                   75                 60                  80 %
Politics                                   106                 79               74.53 %
Video Production                           122                104               85.25 %
Anime & Manga                              239                160               66.95 %
Magento                                    764                223               29.19 %
English Language Learners                  570                462               81.05 %
Sustainable Living                          35                 20               57.14 %
Tridion                                      1                  0                   0 %
Reverse Engineering                         47                 25               53.19 %
Arduino                                    983                885               90.03 %
Network Engineering                         65                 42               64.62 %
Open Data                                   54                 17               31.48 %
Freelancing                                 89                 63               70.79 %
Blender                                     91                 51               56.04 %
MathOverflow                              1635               1355               82.87 %
Space Exploration                           54                 21               38.89 %
Sound Design                                71                 56               78.87 %
Tor                                        109                 93               85.32 %
Pets                                       172                135               78.49 %
Amateur Radio                               32                 29               90.63 %
Italian Language                             5                  0                   0 %
Stack Overflow em Português                399                157               39.35 %
Aviation                                   510                422               82.75 %
Ebooks                                     137                117                85.4 %
Beer, Wine & Spirits                        60                 51                  85 %
Software Recommendations                   321                212               66.04 %
CS50                                         2                  0                   0 %
Expatriates                                231                190               82.25 %
Mathematics Educators                       12                  8               66.67 %
Earth Science                               43                 25               58.14 %
Joomla                                     191                161               84.29 %
Data Science                                74                 51               68.92 %
Puzzling                                   325                241               74.15 %
Craft CMS                                   56                 41               73.21 %
Buddhism                                    52                 39                  75 %
Hinduism                                   128                 67               52.34 %
Community Building                         165                145               87.88 %
Startups                                   250                185                  74 %
Worldbuilding                              650                575               88.46 %
スタック・オーバーフロー                       50                 43                  86 %
Emacs                                        3                  2               66.67 %
History of Science and Mathematics          14                  6               42.86 %
Lifehacks                                  180                105               58.33 %
Engineering                                 55                 28               50.91 %
Coffee                                      50                 36                  72 %
Vi and Vim                                   3                  0                   0 %
Music Fans                                  57                 35                61.4 %
Woodworking                                113                 93                82.3 %
CiviCRM                                    150                117                  78 %
Health                                     892                558               62.56 %
Stack Overflow на русском                 1135                620               54.63 %
Русский язык                                57                 34               59.65 %
Mythology & Folklore                        37                 33               89.19 %
Law                                        161                101               62.73 %
Open Source                                 33                 23                69.7 %
elementary OS                               14                 12               85.71 %
Portuguese Language                          5                  3                  60 %
Computer Graphics                            8                  5                62.5 %
Hardware Recommendations                    25                 13                  52 %
Stack Overflow en español                  327                 49               14.98 %
3D Printing                                579                490               84.63 %
Ethereum                                   270                115               42.59 %
Latin Language                              12                  8               66.67 %
Language Learning                            7                  6               85.71 %
Retrocomputing                              20                 15                  75 %
Arts & Crafts                               72                 62               86.11 %
Korean Language                              2                  1                  50 %
Monero                                      65                 39                  60 %
Artificial Intelligence                     86                 81               94.19 %
Esperanto Language                           8                  0                   0 %
Sitecore                                    16                  0                   0 %
Internet of Things                          49                 23               46.94 %
Vegetarianism                               58                 45               77.59 %
Ukrainian Language                           1                  0                   0 %
DevOps                                       4                  1                  25 %
Bioinformatics                               4                  2                  50 %
Computer Science Educators                  21                 13                61.9 %
Interpersonal Skills                       698                638                91.4 %
Augur                                        5                  3                  60 %
3
  • 3
    83% helpful? We're not declining more than that? unacceptable Commented Mar 9, 2018 at 2:40
  • 1
    It's not entirely obvious that moderators are supposed to flag as spam instead of just deleting, so I'd expect some mods to delete spam instead of spam-flagging it again. Is destroying users as spammers (but not spam-flagging their posts before that) counted properly here? Commented Mar 9, 2018 at 12:23
  • They would be counted in the first table but not the second, @mad
    – Shog9 Mod
    Commented Mar 9, 2018 at 14:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .