57

At present, when something is flagged as "not an answer" or "very low quality", that flagged post appears alongside all other flags that moderators have to handle. However, a little while ago these started being fed into the "Low Quality Posts" review queue and the community gained the ability to handle the flagged posts via votes.

I've been watching this, and the community has done a good job in processing these flags (at least on Stack Overflow). It is therefore my suggestion that these flags be removed from the moderator flag queue and only presented to community reviewers in the Low Quality Posts queue.

We will still be presented with system-generated disputed low quality review: Post has a good score but received delete votes flags for highly voted posts that the community could not remove in review, and disputed low quality review: Undeleted by author for the times where the author overrides the community deletion votes. This lets us act on the rare cases where we do need to step in.

I see very little signal in the disputed low quality review: Controversial review: more delete votes than 'looks good' flags resulting from divided Low Quality Posts reviews, so as a related request I suggest those no longer be presented to moderators.

I would make one modification for how "not an answer" and "very low quality" flags are voted on: if these flags are unanimously voted against, mark the flags as "declined" instead of "disputed". I've been seeing some people shotgun flags of this type, since they know that there is no penalty for getting a disputed flag, and currently community votes cannot outright decline flags. There should be some slight deterrent to prevent people from flagging everything in sight in an attempt to farm helpful flags. Anything other than a unanimous rejection of the flag would still only dispute the flag, but I'd like to see the community be able to decline really bad flags.

Removing these flags would better focus the moderator flag queue on the items only we can handle and reduce disputes with moderators over declined flags (it's harder to argue with the votes of your peers). I think we can trust the community with these flags.

Improve this question
12
  • 10
    This may make sense on larger sites, but on smaller sites I rely on those reports of disputed reviews. Sometimes it really is controversial (and I'll bring it to chat or meta); other times there just weren't enough people with the VtD priv and it didn't stay in the queue long enough to gather six votes from others but it should be deleted.
    – Monica Cellio
    Commented Nov 6, 2014 at 17:02
  • With the time it has been taking to have custom-reason flags handled, this sounds like a very good idea.
    – Mooseman
    Commented Nov 6, 2014 at 17:03
  • @Mooseman There will always be a delay on SO with custom flags, we avg 1-2k flags most days that need to be handled.
    – Taryn
    Commented Nov 6, 2014 at 17:04
  • 9
    This is overdue for implementation. On smaller sites these flags are often the only ones not processed, and, the OCD need to 'process all flags' and keep the 'flag' icon off the toolbar means that many moderators just process the low-quality flag anyway, and thus there's no need for the review queue anyway. it is very hard for me to see the flag notification, and say 'OK, that flag can stay, the community should deal with it.'.
    – rolfl
    Commented Nov 6, 2014 at 17:06
  • @rolfl I agree, if it gets in front of me why would I skip it when I can easily process the flag dropping the XXXL count to XXL.
    – Taryn
    Commented Nov 6, 2014 at 17:07
  • 4
    This sounds like a reasonably good idea, but please don't pull them completely from the mod queue. Add a checkbox or a tab or something so that when/if the community is having trouble keeping up we can just turn it back on, clean it out and turn it back off. Going through a random queue is terribly slow and inefficient compared to the mod queue. If I have little time I can usually still snipe 3-5 of the simpler ones in a minute or so, whereas a queue would take quite a bit more time, especially if it decided to give me a longer/harder one.
    – ɥʇǝS
    Commented Nov 6, 2014 at 17:22
  • 4
    Friendly amendment: where you say "hide NAA/VLQ", instead say "hide flags that are currently in the review queues". (And yes, that suggests a different implementation.) But I also agree with upside-down-Seth that on smaller sites we want to be able to keep an eye on these; a toggle, or just putting them at the bottom of the flag page (labelled), would do nicely there.
    – Monica Cellio
    Commented Nov 6, 2014 at 17:25
  • It would be nice if this was also added at the same time: Can we get a visual indication that there are spam or offensive posts in the flag queue?
    – ɥʇǝS
    Commented Nov 6, 2014 at 17:29
  • @MonicaCellio Even on AU (one of the bigger sites) I'm not sure this would entirely work. We've already had 112 LQP reviews today plus around ~30-40 additional moderator handled NAA/VLQ flags and it's only 9:30 in the morning! (granted, the new day starts at 5:00, so it's not as bad as the words would make you think)
    – ɥʇǝS
    Commented Nov 6, 2014 at 17:31
  • 1
    @MonicaCellio what would be your take on giving these flags a certain (site specific) timeout before displaying to moderators? So that LQ reviewers would have fair chance to deal with flagged post before it gets pushed to moderator attention (side note moderators, as well as 10Kers can browse LQ queue at will)
    – gnat
    Commented Nov 6, 2014 at 20:49
  • 2
    @gnat a delay would be fine, possibly even ideal. I just want to make sure that mods retain the ability to deal with flags that are just languishing because the community doesn't (or can't) act on its own. (BTW yes we can use the review queues, but you get entries one at a time in random order, while the flag page gives us all the flags on one page and it's easy to scan. Also, it reports timestamps for the flags, so I can notice that something has been lingering for 3 days or whatever -- the review queue doesn't tell me if it's 5 minutes old or 5 days old.)
    – Monica Cellio
    Commented Nov 6, 2014 at 20:53
  • 3
    @MonicaCellio agree, that was my concern as well. Them SO mods, with their 30,000 army of LQ reviewers "eating" flags in a matter of minutes, can't even imagine how it could be differently... but we know it could :)
    – gnat
    Commented Nov 6, 2014 at 20:56

2 Answers 2

Reset to default
34

Part of this is implemented, although not quite in the way you suggested:

Very Low Quality and Not An Answer flags do not enter the moderator queue for 15 minutes after they're raised. This applies network-wide, except on Stack Overflow, Mathematics, TeX, Salesforce and Stack Overflow em Português (where the delay is a full hour) and meta sites (where they enter the queue immediately).

This delay gives the community a chance to handle these flags first. If that doesn't happen, or if it can't happen, then it is made available to moderators for resolution. Keep in mind, not all posts can be processed via review, and when a post is processed through review but continues to garner flags the system reserves these for moderators. Therefore, when you do see these flags in the mod queue, you should handle them - it's unlikely anyone else will.

The other two requests - declining some flags instead of disputing them, removing some auto-flags - have merit, but require some additional thought; please raise these as a separate feature-request.

Appendix: low quality handling stats, by site

The following table shows the number of review tasks created during the past 90 days, the % of those tasks successfully handled by review, and the respective median handling times, in minutes, for reviewers and moderators:

Total PctSuccessful Median review handling time Median mod handling time   Site Name 
----- ------------- --------------------------- ------------------------ - --------- 
86850 76.92 %       34                          25                       * Stack Overflow
3928  33.45 %       68                          48                         Super User
1093  36.41 %       40                          13                         Server Fault
22    18.18 %       411.5                       96                         Stack Apps
457   20.35 %       1113                        107.5                      Cooking   
300   55.00 %       84                          32                         Home Improvement
186   27.42 %       164                         93                         Game Developers
2419  36.17 %       115                         46                         Gaming    
539   36.18 %       54                          23                         GIS       
7580  83.71 %       6                           23                       * Mathematics
267   25.47 %       76                          134                        Photography
403   39.21 %       23.5                        68                         Statistical Analysis
465   17.20 %       161                         93                         Web Apps  
118   56.78 %       23                          21                         Webmasters
1901  35.46 %       53.5                        22                         Apple     
48    18.75 %       156                         281                        Theoretical Computer Science
2239  56.10 %       191.5                       141                        English Language and Usage
130   22.31 %       172                         67.5                       Personal Finance and Money
158   18.35 %       57                          34                         Role-playing Games
591   88.32 %       36                          30                       * TeX - LaTeX
4718  36.37 %       73                          71                         Ubuntu    
1171  57.05 %       20                          34                         Unix and Linux
464   35.34 %       42                          27                         WordPress 
104   20.19 %       975                         221                        Bicycles  
810   17.16 %       98                          73                         Programmers
2083  13.68 %       1172                        182                        Android Enthusiasts
78    37.18 %       2097                        2194                       Board and Card Games
879   44.03 %       96                          73.5                       Physics   
34    17.65 %       3199                        3223                       Homebrew  
516   30.23 %       243.5                       131                        IT Security
38    13.16 %       727                         103                        Writers   
539   28.01 %       37                          28                         Electronics and Robotics
360   39.44 %       163                         136                        Graphic Design
428   39.95 %       230                         116                        Database Administrators
736   54.21 %       93                          67                         Science Fiction
126   33.33 %       31.5                        9                          Code Review
257   66.54 %       11                          113                        Code Golf 
70    55.71 %       151                         79.5                       Quantitative Finance
26    26.92 %       1179                        1073.5                     Project Management
131   14.50 %       163                         90                         Skeptics  
524   41.22 %       84                          51                         Drupal Answers
56    10.71 %       2379                        663.5                      Fitness and Nutrition
174   18.39 %       462                         188                        Motor Vehicle Maintenance and Repair
99    10.10 %       265.5                       162                        Parenting 
630   47.46 %       20                          45                         SharePoint
172   41.86 %       306                         167                        Musical Practice and Performance
76    21.05 %       6322.5                      1062                       Software Quality Assurance and Testing
221   52.49 %       89.5                        92                         Jewish Life and Learning
48    22.92 %       697                         1202                       Astronomy 
147   57.82 %       37                          73                         German Language and Usage
52    55.77 %       110                         34                         Japanese Language and Usage
36    63.89 %       197                         55                         Gardening and Landscaping
89    13.48 %       3512.5                      828.5                      Philosophy
24    8.33 %        2434.5                      502                        Personal Productivity
327   29.05 %       134                         279                        Travel    
113   27.43 %       1150                        473                        Cryptography
69    33.33 %       135                         613                        Signal Processing
64    53.13 %       363                         132                        French Language and Usage
516   6.78 %        142                         93                         Christianity
150   21.33 %       447.5                       419                        Bitcoin   
40    22.50 %       464                         124                        Linguistics
77    1.30 %        27                          3                          Biblical Hermeneutics
23    21.74 %       2008                        3209                       Economics 
128   53.91 %       247                         209                        History   
35    20.00 %       392                         377                        LEGO®     
94    28.72 %       4718                        1323.5                     Spanish Language and Usage
22    4.55 %        719                         90                         Computational Science
353   44.76 %       243.5                       200.5                      Movies    
193   63.73 %       471                         3708                       Chinese Language and Usage
275   30.18 %       189                         158.5                      Biology   
21    47.62 %       414.5                       269.5                      Poker     
312   56.41 %       20                          52                         Mathematica
53    15.09 %       271.5                       295.5                      Cognitive Sciences
50    20.00 %       141                         193                        The Great Outdoors
25    12.00 %       2160                        433.5                      Martial Arts
64    37.50 %       191.5                       178.5                      Sports    
206   33.01 %       209.5                       59                         Academia  
133   36.09 %       141                         106                        Computer Science
155   21.29 %       748                         322                        The Workplace
214   64.49 %       514                         531                        Windows Phone
221   47.51 %       132                         154                        Chemistry 
63    39.68 %       1235                        1707                       Chess     
182   11.54 %       573                         51.5                       Raspberry Pi
115   60.87 %       402.5                       379                        Russian Language and Usage
213   11.74 %       2594                        165                        Islam     
512   60.35 %       91                          138                      * Salesforce
53    35.85 %       307                         464                        Patents   
170   21.18 %       273                         64                         User Experience
12    0.00 %        null                        238                        Genealogy and Family History
23    8.70 %        6313                        5049.5                     Robotics  
63    34.92 %       950                         1049                       ExpressionEngine
35    8.57 %        84                          180                        Politics  
34    14.71 %       2644                        133                        Audio-Video Production
285   43.51 %       171                         137.5                      Anime and Manga
904   47.90 %       79                          39                         Magento   
397   49.37 %       64                          141                        English Language Learners
12    41.67 %       1027                        73.5                       Sustainable Living
28    17.86 %       418                         47                         Tridion Stack Exchange
35    25.71 %       4587                        1311                       Reverse Engineering Stack Exchange
84    27.38 %       489                         122.5                      Arduino Stack Exchange
89    26.97 %       390                         141                        Network Engineering Stack Exchange
21    33.33 %       429                         374.5                      Open Data Stack Exchange
14    7.14 %        30                          784                        Freelancing Stack Exchange
188   47.87 %       100                         103                        Blender Stack Exchange
346   32.66 %       40                          43                         MathOverflow
84    34.52 %       95                          25                         Space Exploration Stack Exchange
54    20.37 %       159                         92.5                       Tor Stack Exchange
53    15.09 %       337.5                       238                        Pets Stack Exchange
25    4.00 %        214                         1667                       Amateur Radio Stack Exchange
19    15.79 %       166                         315                        Italian Language Stack Exchange
546   60.07 %       41.5                        26                       * Stack Overflow em Português
118   56.78 %       327                         157                        Aviation Stack Exchange
18    11.11 %       2046.5                      853                        Ebooks Stack Exchange
19    21.05 %       7112.5                      1515                       Beer Stack Exchange
156   14.74 %       265                         153                        Software Recommendations Stack Exchange
30    23.33 %       158                         154                        Expatriates Stack Exchange
21    28.57 %       117.5                       92.5                       Mathematics Educators Stack Exchange
22    22.73 %       314                         221                        Earth Science Stack Exchange
66    25.76 %       66                          604.5                      Joomla Stack Exchange
25    32.00 %       1102                        656                        Data Science Stack Exchange
345   44.93 %       128                         77                         Puzzling Stack Exchange
50    50.00 %       81                          136                        Craft CMS Stack Exchange
56    32.14 %       401.5                       331                        Buddhism Stack Exchange
43    18.60 %       85.5                        49                         Hinduism Stack Exchange
2     0.00 %        null                        88.5                       Moderators Stack Exchange
23    21.74 %       251                         114.5                      Startups Stack Exchange
134   32.84 %       359.5                       164                        Worldbuilding Stack Exchange
74    28.38 %       134                         70                         スタック・オーバーフロー
70    50.00 %       69                          64                         Emacs Stack Exchange
Improve this answer
12
  • 9
    Disagree with "unlikely anyone else will". Even in the fast-moving LQRQ on Mathematics, many items take longer than 15 minutes to process -- after all, it takes 6 reviewers to go into the queue and recommend deletion. A typical example: nobody doubted the post should be deleted, but it took > 30 minutes to get 6 RecDel votes.
    – user259867
    Commented Jan 27, 2015 at 20:38
  • 11
    I'm open to expanding this on individual sites (as indicated by Stack Overflow's hour delay) - but they have to be at a point where the community actually is handling a majority of flags reasonably quickly. FWIW, Math processes over 50% of LQ review tasks in about 6 minutes; an hour delay would put them at 92%. By contrast, The Workplace requires a whopping 12 hours to clear 50% of LQ review tasks; adding a longer delay there would just leave crap festering longer until the mods handle it anyway.
    – Shog9
    Commented Jan 27, 2015 at 20:42
  • Thanks for the stats & site-specific extension. So, on Robotics it takes 3.5 days for a mod to handle the median LQ task... weird.
    – user259867
    Commented Jan 27, 2015 at 22:47
  • Not really; they get about one of these posts every 4 days, so it's not like they're sitting by the proverbial phone waiting for a call.
    – Shog9
    Commented Jan 27, 2015 at 22:49
  • I think this is brilliant.
    – davidlowryduda
    Commented Jan 27, 2015 at 23:32
  • 1
    What does "successfully handled by review" mean exactly?
    – ɥʇǝS
    Commented Jan 28, 2015 at 2:43
  • Pretty much what it sounds like, @ɥʇǝS - the review task was completed by reviewers. As opposed to invalidated due to actions outside /review (post being deleted by its author, moderator handling the flag, upvotes / closing / locking making it ineligible for further review, etc.)
    – Shog9
    Commented Jan 28, 2015 at 2:46
  • So would moderators who handle things while inside the review queue be under the "review handling time" rather than "mod handling time"? Or both?
    – goldPseudo
    Commented Jan 28, 2015 at 5:38
  • Both, @goldpseudo
    – Shog9
    Commented Jan 28, 2015 at 5:42
  • I've noticed that when I filter for only "very low quality" in the flag queue even new ones appear. Is this a bug or expected behaviour?
    – user642796
    Commented Jan 28, 2015 at 17:49
  • Seems reasonable, @Arthur.
    – Shog9
    Commented Jan 28, 2015 at 17:50
  • Does the delay apply also to VLQ/NAA flags that are not eligible for LQRQ processing? I see the sentence "If that doesn't happen, or if it can't happen, then it is made available to moderators..." but it's not clear if "or if it can't happen" means a special clause that speeds up the flag.
    – user259867
    Commented Mar 22, 2015 at 20:40
11

Idea to give LQ reviewers better chance to complete review makes perfect sense to me, especially given that SE team has built a pretty decent safety net to handle possible distortions.

Proposed implementation ("no longer be presented") though seems to be missing that flagged posts can simply stuck in the review queue. This is highly unlikely at a larger site like Stack Overflow but at smaller sites, it can easily happen.

I think that it would be more reliable to have some delay before pushing flags for moderator attention. Per my experience at different sites, I would say that 2-3 hours delay look reasonable, but this would better be checked with stats. Maybe it would be better to even have this delay to be configurable per site, to account for possible differences in review flow.

I also support the idea to introduce flagger accountability for disputed flags. I can't tell if parameters to trigger suspension should be the same as for mod declines, but flaggers who consistently and intensively fail, say, half of LQ reviews, would better be throttled and given a better chance to find out and learn about this.

I am not sure though that there is a need to re-qualify disputed flags into declined for above. Worth also noting that as a flagger, I find it convenient to study my flagging mistakes when I can easily tell which ones come from moderators and which ones come from LQ reviewers (FWIW in my experience disputed flags typically indicate cases where I made "larger" mistakes).

Improve this answer
2
  • 1
    To your time limit, looking at the 100 "not an answer" flags currently being presented to moderators on Stack Overflow, a large number are up to 12 hours old, so 2-3 hours might be too little of a delay to give the community time to handle them. Only a handful are older than 12 hours, although that might be colored by some recent flag cleanups certain moderators have been doing.
    – Brad Larson
    Commented Nov 7, 2014 at 21:07
  • 1
    thanks @BradLarson - as I suspected my guesstimate seems to somehow miss: "this would better be checked with stats" :)
    – gnat
    Commented Nov 7, 2014 at 21:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .