78

Many hands make light work.

I propose that we offer a coveted Stack Exchange coffee mug to the top 25 people who work to clear the close queue in the next 45 days, and a USB rocket launcher to the top reviewer.

enter image description here(image from Hyperbole and a Half)

This will do two things.

  1. It will clear the close queue. Yay!
  2. It will renew interest in community moderation, which I feel has been lagging a bit lately.

Accuracy of reviews can be assured by the degree of concurrence with other reviewers. No SE programming is required, only a few database queries.

40
  • 18
    Coffee mugs! Those will work great for all that coffee I never drink!
    – animuson StaffMod
    Sep 27, 2013 at 17:17
  • 15
    @animuson: Whiskey can be drunk from coffee mugs as well.
    – user1228
    Sep 27, 2013 at 17:26
  • 9
    What if the top 25+ is tied by 25 people having done 45 * 40 reviews? Sep 27, 2013 at 17:28
  • 10
    Moderators have no review limits. Disqualified?
    – Ry-
    Sep 27, 2013 at 17:31
  • 12
    Hi, I’d like to resign, please…
    – Ry-
    Sep 27, 2013 at 17:32
  • 9
    @minitech: If you clear 50 per day for the next 45 days, I'll send you a mug myself.
    – user102937
    Sep 27, 2013 at 17:32
  • 4
    @animuson That's the mod mug. It's got a hammer and a scythe on it.
    – user102937
    Sep 27, 2013 at 17:34
  • 2
    I need another 1300 rep on SO NOW. Mugs are at stake, people!
    – Mike G
    Sep 27, 2013 at 17:40
  • 3
    @RobertHarvey "You need at least 3k reputation to review Close Votes." From stackoverflow.com/review. My current rep on SO: ~1700. Unless there's some subtlety that I missing here.
    – Mike G
    Sep 27, 2013 at 17:44
  • 3
    @RobertHarvey If the close review queue only needed 500 rep, we probably wouldn't have this problem!
    – Mansfield
    Sep 27, 2013 at 17:59
  • 31
    We would have... A different problem. @Mansfield.
    – Shog9
    Sep 27, 2013 at 18:53
  • 4
    Does anyone have any estimate as to how many questions come in on a daily basis that deserve to be closed but don't even receive one vote-to-close because people figure "why bother"? I'm just curious if this queue-clearing is being viewed as a "solution" or is just one step in a more holistic approach to addressing the close process? Sep 27, 2013 at 21:04
  • 5
    @PeterAlfvin: The purpose of review queues is to surface posts to trusted users that have an interest in keeping the site clean. It just takes one vote to put a question into the close queue. This works extremely well on other sites such as Programmers. However, it's hard to take that queue seriously on Stack Overflow when there are 80,000 posts in there.
    – user102937
    Sep 27, 2013 at 21:08
  • 7
    I would be reviewing posts, but I was banned due to this crazy audit.. stackoverflow.com/review/reopen/2976851 Sep 28, 2013 at 1:16
  • 4
    Actually, @JoshC, you were banned for failing this one.
    – Shog9
    Sep 28, 2013 at 15:23

11 Answers 11

37

No one can win this contest. The limit of 40 reviews per day will cause approximately one bazillion of us to arrive in a dead heat.

7
  • 13
    Lift limit for duration of contest? Sep 28, 2013 at 1:07
  • 1
    The first [x] to arrive at the highest number could receive the prize, then. @user2162192, I think lifting the limits would probably cause more problems than it's worth, but that's just my $0.02.
    – WendiKidd
    Sep 28, 2013 at 1:39
  • 3
    Isn't this a good problem to have? Anyway, an accuracy factor can be used to break the tie for first.
    – user102937
    Sep 28, 2013 at 2:25
  • I doubt that there will be less that .5 bazillion people with identical accuracy ratios. Most of the close queue is not exactly a collection of subtle questions.
    – Rosinante
    Sep 28, 2013 at 18:54
  • I don't know the rate of backlog of the review queue, but just 50 people doing the maximum review rate for the duration of the contest would more than clear the current close votes. If there are 25 mugs at stake, the queue would stand no chance of staying that full very long. Oct 2, 2013 at 0:10
  • 1
    and yet the queue gets larger and larger.
    – Rosinante
    Oct 2, 2013 at 0:15
  • Maybe first one to add 1800 reviews?
    – user206222
    Oct 18, 2013 at 0:17
21

I've all but given up on the close votes queue on Stack Overflow. It's psychologically damaging attempting to make a dent as you can't tell whether you actually are or not.

At the time of writing this question has 39 upvotes and 226 views. 39 people can make a dent of around 17.5k in the queue over 45 days, which'll keep it under control. 226 can clear it completely and deal with any increase.

Based on the assumption that people are going to read this answer, I'm going to make a start. I promise to use up all my close votes every day I have access to a computer for the next 45 days - mugz or no mugz (though I do need some!). We don't need anything special to do this, we just need to get over the fear.

Join in and let's beat the people polluting the site or save those who aren't.

Credit to uncyclopedia.wikia.com/wiki/File:Cat_using_computer.jpg

Close all the things!

P.S., the people who win the mugs will be the people who vote to close the most questions get deleted the same day; giving people with more than 20k an unfair advantage...

1
  • 4
    Views and upvotes means nothing - many of us cannot review, or can, but only on other sites in the network :(
    – Mołot
    Oct 2, 2013 at 11:11
10

The current queue is 82.2K

It is growing at roughly 10K per month (observational, may not be accurate)

So you're looking at about 97K in 45 days (let's rough up to 100K)

So at 45 days, 50 reviews per person, that's only 222 (5 votes per post) people beyond what is normally closed in a given day. I'd say it is definitely doable.

(And this doesn't even take into account the fact that many posts will have close votes already - assuming an average of 2 existing close votes, that brings the number of people down to 133 of 17,000+ eligible close voters).

5
  • We're not allowed 50 reviews per person per day. Only 40.
    – Rosinante
    Sep 28, 2013 at 0:53
  • 3
    Lift limit for duration of contest? Sep 28, 2013 at 1:08
  • @Rosinante speaking for yourself, I have 50 when i go to the review queue :-) Sep 28, 2013 at 1:09
  • 8
    I have 50 close votes, but the review queue does the 'thanks for reviewing 40 questions, come back in 24 hours' thing. I can then close others not in the queue.
    – Rosinante
    Sep 28, 2013 at 1:11
  • 1
    We're down to 81.9k already and it hasn't even been a day :D
    – Doorknob
    Sep 28, 2013 at 15:46
6

Firstly, I am a lo-o-ong way from being able to participate in closing posts. I am still something of a noob around here.

Secondly, I have wa-a-ay too many mugs.

In my thinking, the system got like this because of a flaw in the system design, so short term fixes will be just ... short term.

The key change is to automate more of the close process. From my observation many questions on the queue (how many?) have no answer marked correct AND the question is down voted. So I suggest to auto-close

  • All posts with > 2 close votes AND no answer marked correct AND the question is down voted AND question is > 1 year old.

  • All posts with > 3 close votes AND no answer marked correct AND the question is down voted AND question is > six months old.

This change would close the (mostly) no-brainers, and it still relies on the community to flag and down vote posts. The exact parameters, such as the question age, can be fine-tuned for optimum results. And the change should be phased in with gradually tightening criteria over six months, so that mods are not swamped by complaints.

Auto-closed posts should marked as such, so that the OP knows why and can challenge the closure (pun).

10
  • So how would a close reason be chosen? Oct 25, 2013 at 3:53
  • 1
    Having Community ♦ come in and autoclose some old bad stuff sounds ...not too bad. Keep in mind close-vote aging, though - after a question reaches 100 views, close votes expire after four days. Also, what happens when a year-old downvoted question gets to two votes - does Community ♦ just go and seal it immediately? It's been stated by SE staff that the five-CV threshold was to prevent "a couple of mistakes" from closing a question. This proposal would give "a couple of mistakes" on an old question moderator-bot approval. (Minor objections, though - overall, IMO, good idea.) Oct 25, 2013 at 3:58
  • @LanceRoberts In agreement with the voters if there's a majority; if there isn't (and it can happen normally, with five people all picking a different reason), first voter wins. (Jeff said somewhere that that's how the reason display chooses already.) Oct 25, 2013 at 4:00
  • @LanceRoberts "This question was automatically closed by the CloseBot. See the FAQ." Thanks to michaelb958 for the Bot idea.
    – andy256
    Oct 25, 2013 at 4:01
  • @andy256 Not even a new bot - we already have a bot to do janitorial things. Oct 25, 2013 at 4:03
  • @michaelb958 In principle the five vote concept has merit, but we appear to have found a gap. I didn't know (mea culpa) that close votes expire. Hmmm ...
    – andy256
    Oct 25, 2013 at 4:05
  • Here's a mockup. Oct 25, 2013 at 4:12
  • @michaelb958 Nice. Do you know (or how can I find out) if expired votes are removed or the close algorithm applies the aging? My reading of the schema doesn't tell me.
    – andy256
    Oct 25, 2013 at 4:22
  • @michaelb958 To answer my own question, they get deleted. To answer your earlier question then, my proposal just lowers the threshold for closure.
    – andy256
    Oct 25, 2013 at 4:52
  • @andy256 Overall, I think I like this idea. Oct 25, 2013 at 4:55
4

How about no reward at all?

Temporarily drop the rep limit to 2,000.

Provide notes on what is expected of a Close Review.

Allow people who have a mind to to just plough into them (in a considered way, after reading the notes provided).

Nothing towards badges. Nothing towards anything. Close Review returns to 3,000 once some manageable number are outstanding.

No incentive for robo-anything. Simple mistakes in review are reduced in effect through requirement for multiple votes anyway. The standard of the notes provided (pops up when hovering somewhere, or a click, or whatever) reduces the number of simple mistakes.

If going ahead with any strategy, some investigation into what the major parts of the problem are, and which type of questions (if such can be decided) should be tackled first would be a big help, and motivational to see the length of the queue start dropping at the beginning of the task.

And not to forget, analysis of how it came about in the first place, and remedial action on that. I've seen a comment that the backlog existed, just was not exhibited, before the Review Queue system existed. Fair enough. But that does not explain 25,000+ increase in the last eight months or so. Or does it?

Without something to stop it happening again, it'll happen again. Then expecting The Community to do anything about it will be really demotivating.

4
  • 3
    Some adventurous user could create a query in the data explorer seeing how many people are regular reviewers of suggested edits, etc. but don't have 3000 rep for close votes yet. Judging from the number of people, you could even get a nice estimate for how much this would help. If you furthermore checked their accuracy on suggested edit reviews compared with the average over a month or something, you could even argue how good they would be at reviewing!
    – jmac
    Oct 18, 2013 at 12:44
  • @jmac, yes, I like that. Something to measure whether it would be worth it. The reviews that I have access to are available only for seconds, so there are enthusiasts. Are they all going just for badges? All doing a bad job? Oct 18, 2013 at 12:49
  • You can create a javascript query using the suggested edits API that checks the past month, reputation of the reviewer, and index by what % agree with the edit by reputation (3000+ vs. 2000+). Alternatively, you can use the data explorer if you know TSQL or are competent in that realm. I am good at neither, hence the suggestion for an adventurous soul to do it.
    – jmac
    Oct 18, 2013 at 12:52
  • Mmm... I'm no good at them either (actually, I've never tried them, so I can't be certain). I can't find @an adventurous soul, I'm sure they would have been a good choice, but gone now. Perhaps someone else... Oct 18, 2013 at 13:01
4

I think a contest is the right approach, but it needs to be done in a big way.

First, set an amount of time, I was thinking two months, though 45 days is fine. If you do that starting next month, you can end things at the end of the year, but that's not crucial.

Second, you give out one-time permanent badges for the contest commemorating their achievement forever. Let's say for 250, 500 and 1000 closes in that timeframe (obviously you should probably remove the review limit for the duration of the contest).

Third, you give out either rep or even specialer one-time ranked permanent badges for the top 3 closers.

Make it as big as the Winter Bash Hat thing, go wild.

EDIT: Now that we're into the 2013 Winter Bash, we can really see the incentive even just a temporary award creates. The close queue has really been decreasing. Lets plan for a big closing contest this next year with some of the permanent rewards I mentioned.

2
  • 1
    Do you not remember what happened to the entire review system the last time we added a bunch of new badges. It made the entire review system unusable for a long period of time and caused major damage all throughout the site.
    – Servy
    Oct 17, 2013 at 22:43
  • @Servy, guess we should do it better this time around. Oct 17, 2013 at 22:48
3
+300

I think envolving the comunity in this kind of event is fun and good for the "community feeling". If the queue goes down some 10~20K its already a common victory.

I write this to ask, again, to raise the close limit during the contest. There will still be audits "on", to control the quality (If the audits stop or not Robo-reviewrs is another discussion). This will also be a good opurtunity to test if raising the limit is useful to get down that >80K number by checking the stats after this.

Contest? Yes!
Raise the close vote limit during the contest? Yes!

4
  • It's up 10K since your August 31 question. Even a drop of 20K doesn't seem like a victory in the light of that. By the New Year the queue would be back to about the same as now. There has to be a killer surge, which does not cause more problems than it solves, and which is allied with an analysis of why it happens and corrective action. The owners cannot afford 25 coffee mugs every quarter just to slip more slowly. To own up to my interest, I have a flag that has been stuck in the queue since July. Oct 18, 2013 at 15:25
  • @BillWoodger, there has been many discussions about this. I would be impressed if there would be a 20K drop. Daily voting limit has been pointed to be able to enable more reviews for those who have the rep and time for that.
    – Sergio
    Oct 18, 2013 at 16:02
  • That was the post I was referring to. I'd not be impressed. If 1%+ of the questions on the site are in limbo, and the trend is steady growth, something has to be done. You're right, it is not lack of suggestions, but lack of action afterwards. Do we even have knowledge of why it occurs? If not, the problem doesn't go away, even if more than 20k of the queue does. Oct 18, 2013 at 16:09
  • @BillWoodger, yep. That is why I posted this "answer", then the close vote limit suggestion could also be tested.
    – Sergio
    Oct 18, 2013 at 16:10
2

The problem with the accuracy factor: doing an "edit" will technically mean that everyone disagrees with you. Will there be a special case for editing or something like that?

1

Bad idea for one main reason:

Users are already robo-approving for essentially no reward.

What would happen if you give them actual stuff?

We'd end up with a whole lot of questions that are either open and should be closed, or closed and should be open.

And honestly, I doubt it really matters if you resolve ties based on accuracy or what. Many users will just try regardless. Some will get banned, sure. But will enough get banned quick enough before they do too much damage? Probably not.

Admittedly I don't really know if robo-approvers are a problem on the close-vote queue, if it were, the queue probably wouldn't be so backed-up (although, a few robo-approvers probably won't make such a big difference in the numbers), but, if it isn't, this suggestion might cause it to become a problem.

2
  • Wait, so you're saying we shouldn't do it at all? What is your alternate idea?
    – user102937
    Oct 2, 2013 at 16:08
  • 2
    Giving people stuff for reviewing stuff? No, I don't think we should do that at all (and least not in a sprint-type setting, long term might be able to work though). Tending to the close vote review queue? I put a few ideas out there some time ago, back when no-one else seemed to think it was a problem (though I must say, looking back, some of them don't seem like particularly good ideas). Oct 2, 2013 at 16:11
1

I love the idea spirit. What worries me is possible unwanted side-affects of a contest.

  • Already there are complaints about too many robo-reviewers. I suspect badges have a major part in the robo-reviewer behavior.

I'm afraid that an outright contest may bring out the worst of that kind of behavior.

  • But, I agree that creating motivation is key. With over 18,000 3K+ rep users on SO, if they were motivated to perform an average of 2 review tasks each per day, the close queue could probably be cleared within a couple months.
0

I'd propose another metric to calculate the posse leaders, and the ...
well, the is whatever the staff comes up with.

The idea is: in a extended period of time, say 6 months,
whoever casts the most final close votes, number 5 out of 4, leads,
and I think this should work out of the review queue.

For that we need a query like this: Get unclosed questions with closed votes.
And I'd love to know how to fork it to also filter by tags and body length, gimme teh codez, plz!.

data explorer


Of course this proposal favors me because I'm already doing that :P
of course not, rules start when rules start

Out of 50 daily votes, I spend about 40 in those results.
Average optimistic closure 200/week, 800/month.

IF the Question IS (out of my are of expertise) AND IS (long and/or have code in it), I skip it.
Otherwise, IF:

  • "I have this idea -some explanation- what to do?..., hmmm, donnow

  • What does $foo means in -any language-..., hello?, RTFM

  • Plz, help me -because yes- and gimme teh codez bcz -more yes-..., ok, good luck

  • Recommend me -where is the nearest WalMart- or sheiz..., ...

immediate closure.


Well, more than anything, I wanted to share the 40 daily-close-killer-votes technique.

1
  • And I also love to find those 3 close-voted Q's more than in need of terminating.. So I add the 4th and leave the path clear to the next reviewer to cast the goal.
    – brasofilo
    Oct 25, 2013 at 3:15