Huge close votes review queue on Stack Overflow [closed]

Question

Today, the size of the close-votes queue is 55k questions and growing. Is that a problem?

Which action could be taken to clean it up? It seems to be growing faster than Stack Overflow members are capable to handle it.

Answer 1

Which action could be taken to clean it up?

TL;DR Educate active reviewers to filter SO Close Votes queue by single close reason, with "duplicate" and "too localized" additionally filtered by their favorite tags.

Based on my experience, this leads to substantial productivity gain.

• For example, I think it would make a perfect sense to introduce New bronze badge for using a filter on the Close Votes review queue to raise awareness of this feature.

Boring details on what led me to above conclusion are provided below; feel free to skip these.

---

Initially I've been going through close votes queue unfiltered. This worked just fine in all other queues I tried (dozen or two queues at SO and some smaller sites) and I saw no reasons to do it other way.

This queue felt a bit difficult from the beginning, but I thought I'll get used to it after a while. 300+ review actions passed, but it still felt cumbersome, at this moment I got curious why is that?

• Discomfort I felt when running through queue reminded me of old review system and of the reason why I didn't want to use it: items in queue were all too different and I just couldn't get into flow because of that. This naturally led me to idea to give filters a try.

My testing of filtering by close reasons began with "off topic". Bingo! Review went smooth and fast, and most important, I felt comfortable - just about like I felt with all other queues in new system. I tried "not constructive", then "not a real question" - great, the magic is there.

Testing "too localized" though didn't go so well. It felt cumbersome again - just the same feeling of having to deal with too many distractions. This lead me to idea to try filtering it further. I added a tag I was comfortable with and the magic flow was back again.

Based on my experience with "too localized", I switched to testing "duplicates" filtered by favorite tag from the very beginning. This worked well, too. Well, to be precise, duplicates review went slower (much slower) than the rest: I had to study both questions, and in some cases I also had to check the answers to both questions. But (and this is very big BUT) what is important, it felt really comfortable - as opposed to my prior experience with unfiltered queue.

Answer 2

To reduce the backlog:

define a function: close(p) = ....

where the right hand side includes the number of close votes, the total rep of all the close voters, the net vote score of the post, the number of answers to the post, and the age of the post.

In other words, 3 close votes from high-rep users, on a negative-score question, with no answers, should just close the thing. Or some rule like that. Then we could concentrate on content that had some chance of having value.

Answer 3

Take a minute here for a sanity check:

Update as of 10/24/2013: 86k need review, 1,151k all-time reviews

There's a massive amount of reviewing being done there. It takes 5 voters to close or 3 "do not close" reviewers to kick a question out of the queue without closing. And yet, folks are making some impressive progress.

Frankly, I don't particularly think this needs to be sped up any - closing is hard by-design. If it takes a while to get through the backlog, that's probably ok...

Update: review task creation / deletion over time

In response to LittleBobbyTables' concern, here are some graphs that may help to illustrate the problem with clearing this queue:

Close review tasks created

Close review tasks deleted (either completed in review, closed or deleted elsewhere)

Review tasks are created in response to close votes or flags - these have been going up.

Answer 4

At the moment we can filter by tag and by close reason, but off-topic has 7 sub-close-reasons.

It may improve the workflow a bit to add additional filtering and be able to narrow it down to one or two sub-close-reasons.

Related: Close Vote Review, additionally filter off-topic close reasons

Answer 5

After all this time, I wanted to add a view of why people don't visit the queue -- or -- at least, why i don't visit the queue.

I have a limited number of close votes. I visit the queue, I use them all, and I feel like I'm the bellhop in Hilbert's Hotel. The amount of work to do remains infinite. I have zero feeling of accomplishment.

In my view, the size of the queue is a view into the ongoing torrent of crap dumped onto the site. I'm willing to keep trying to help deal with it, but I need a larger gun if you want me to help shoot this many zombies.

Answer 6

Just as an input to this discussion, a small progress report.

After a couple of weeks (or maybe more? don't have the energy to check) on Stack Overflow spent maxing out my close vote queue reviews filtered by the cocoa and objective-c tag, today I got the very nice message

There are no items for you to review, matching the filter "[cocoa] [objective-c]"

(thanks to @gnat above for attracting my attention to the filter)

A couple of random observations from this marathon clicking exercise, in no particular order:

• It is generally recognised that bad questions are a problem, but bad close votes are nearly worse. They contaminate the close vote queue and take up at least 4 persons' time to read and analyse (especially in the case of duplicates). A mediocre question with no votes and no answers will just sink down to the bottom of the "sea" of questions, while the same forgettable question with a random, half-warranted close vote now (thanks to the review queue) requires 4 or more people's time and effort.

• There is a pattern of questions that are regularly closed, for instance requests for tutorials. I wonder if there is a possibility to catch such questions (based on an automatic filter) and ask the poster to verify that the question is not simply a question of that type. Naturally, if they tick a box, the question will be posted in the normal fashion.

• The review queue definitely does push one into piling votes on top of the first one. After having cast several hundred votes I am sure that has happened to me, too. The point is, I am not even sure it is a bad thing; if you trust the user who cast the first vote and did the work of analysing the question and, in the case of a dupe, searching for the "best" dupe to point to, that is a great help. Maybe it should be harder to cast the first vote (open the close process) and/or easier to participate in the subsequent votes?

• There is a lot of junk out there, and I have to say that I don't subscribe to the idea that if just a single person out there is helped by a question, it is worth preserving. The effort to curate those low-quality, localised questions just doesn't warrant it.

• I have moaned in the past that I thought it was too much to ask the community to wade through more than 50.000 questions in the review queue for no real reason (what is the point in putting on hold a 9 months old question that no one has looked at since the day it got posted?) Now that I have actually done it (for the tags mentioned) I can safely say that it is not just something I think. I am certain: it is largely a waste of time and effort. It does have a certain therapeutic effect, though, a bit like that of a punching ball (you get to close a lot of stupid questions, even if you don't have the option to call them that), but I don't think that was the intention with the queue.

Answer 7

My suggestion: Encourage people who create close votes to also do some reviewing.

To clarify:

When someone votes to close a question, I suggest that the system should check their reviewing records; if they haven't reviewed any close votes recently, then a polite notice should be shown asking if they would consider doing some reviews to shorten the backlog as well as just adding to it.

I would hope that this would encourage more people to do a few reviews who wouldn't do so otherwise.

(the above assumes that the user has permissions to do reviews, of course)

[EDIT]

I see this working a bit like the voting system does now. Right now, if you mainly vote on answers, you might occasionally get a message when you vote on one saying "You haven't voted on a question lately. They also need some love." (or words to that effect).

So how about doing the same with close votes -- if someone's issued a number of close votes without using the review system, they could get a gentle reminder like "You haven't done any reviewing lately. Please help reduce the workload by reviewing some questions with close votes."

It's not intrusive, it only pops up once in a while, and it doesn't force anyone to do anything, but it might just help do something about the backlog.

(by the way, the close vote review queue is growing rapidly at the moment; it's pushing toward 70k. It's gotta be seen as a problem sooner or later, hasn't it?)

Answer 8

Let's take a look at the problem. As of today, this queue is up to 81.8k questions in the queue. Assuming each has 1 closed vote, that means that a total of 320K actions are required to close all of the questions. Given 20 per day, that means that 160K people would have to clear out 20 in order to close the queue. Then after it is closed, then more people need to review them than have been doing so. So, how can these get reviewed? There are a few ways:

1. Increase the number of people reviewing.
2. Decrease the amount of things that need to be reviewed.
3. Increase the number of things that can be reviewed per day per person.
4. Decrease the time per review required

Okay, given these options, how could any of them be done?

Increase number of reviewers?

Close vote reviewing is a painful, thankless task. Sure, it's the easiest badge of the various custodial badges to earn, but a closed vote decision can be quite difficult sometimes, depending on the nature of the question. Perhaps there should be some incentive to encourage more reviewers, either in the form of reputation, or increased privileges.

Decrease the number of things to review

There are a couple of ideas that I have that fit into this category. Either somehow things need to be automated better, or the number of close votes could be reduced. As the latter could be dangerous, perhaps we should just allow certain users to have more closed votes, perhaps those with a very high reputation, or maybe those who have reviewed more closed votes overall. Both help to filter out the relatively new closed voters, and leave those with more knowledge of how the site goes more power to help it out.

Increase the number of things that can be reviewed

This improves the situation so long as you don't burn people out. As the pile is getting deeper, it could be a quick short term solution.

Decrease time per review

Filtering by tag or closed reasons, better displays to help users figure out what's going on, anything to make the task of reviewing easier increases the likelihood that a user will use all of their closed votes per day, and come back tomorrow and use them too.

I'm sure there are other ideas out there, but here's my preferred method:

1. Allow users with a Reviewer Badge to get 2 close votes, and with a Steward to get 3 closed votes (Might need to be tweaked, but I'm putting out something here)
2. Increase the number of reviews for the closed queue, at least for a while.
3. Do some filtering automatically. The top questions I look at should be for tags I am familiar with. This will make the review queue simpler to manage, as I won't have to learn what the question is about before I look into it. I know that I can filter them manually if I want to, but it's not that obvious to a new reviewer, and thus I think it should be done automatically.

Answer 9

Disclaimers: Though I am relatively new as a registered user of the site, I have been using it for a long time and have read a lot of posts that have been closed for various reasons. Also, since I can't yet participate in close-queue reviewing, I certainly can't speak of how easy or hard it is to do so.

Boring argumentation

Most propositions made here discuss how to deal with the close-queue as it is now. But it is even important to reduce the number of questions that do enter the queue in order to eventually reach the end and then keep up without tremendous efforts.

Now, a quick look at recent questions that were voted to close shows that a large number of these questions come from users with no reputation or almost none. A person learning a new language or technology, or even learning programming for the very first time, either at school or by following whatever tutorial found somewhere, is certainly the archetype of these users.

I don't think it is pertinent to discuss if these persons are being too lazy. It is obvious, to us at least, that the answer they need is lying somewhere around. But when you know very little, it might be hard to decode answers that are not tied exactly to your situation. So they come here, expecting to receive help. They are not considering the fact that their questions should provide common, persistent knowledge for others. And even if they would consider that aspect, their view of "knowledge worth sharing" is certainly tainted by the fact that they indeed have a very limited knowledge of the technology being learnt. Is it bad that they come asking for help? I don't think so. But their questions and the associated answers are certainly not assets to the Stack Overflow community.

So how does all that might help the present discussion? It basically tells us where to put efforts in order to reduce the influx of to-be-closed questions. These persons are looking for answers cut precisely to their needs, which they won't find anywhere. Stack Overflow is very popular and very well known for offering quick answers to questions being asked. Add those two facts, and you realize that you can't expect these users to stop coming here, nor can you expect them to stop asking low-quality questions. Not providing answers to these questions will not discourage students from coming here. But having a few persons trying to provide a quality answer (the kind that guide the user toward better understanding, rather than the kind that is to be copy-pasted), while five persons are reviewing the very same question to vote a close is certainly a waste of everyone's time.

So here are my suggestions:

1. Have users posting questions flag by themselves if their question is of the "homework-or-of-little-interest-to-the-rest-of-the-community" type. Flagging as "homework" should not prevent posting a question (otherwise, people will simply repost without flagging, so we would be back at square number one). This question should be very prominent for first time posters, then could be replaced by a subtle checkbox once users have posted a few questions that has received community support (let's say vote up on two questions, or valid answers on at least five not-closed questions).

2. Define a review-before-show-up-queue, where questions from low reputation users are sent before being officially displayed. Accepting a question from that queue would require much less votes (let's say two, at most) than closing an existing post, and thus, require much less work from the community. Acceptance of posts should not be based on the fact that the question is or is not a duplicate, but rather that the post is correctly flagged as "homework" if it appears to be so, that it actually is a question, that the question demonstrates some effort in understanding the problem (that is, the question does not appears to be a copy-paste from the problem statement), that the pertinent material, and only the pertinent part, is provided (existing code is there, but not a whole project), etc. Actually, all these expectations should be displayed upfront to first-time posters, with checkmarks on each, so that they can provide their questions in the correct format upfront.

3. Add a special icon on "homework-style" posts in all questions lists. Or allow users to hide them. As a last resort, it could even be considered to move these questions to a distinct site. Something along the line of "programation-coaching-exchange". Anyhow, users that spend time answering questions should decide for themselves if they do want to spend time coaching beginners, and guests coming to the site should view higher quality questions, not beginners questions, so they think of Stack Overflow as a place for quality knowledge, not for Computer 101 course complement.

4. Add a reminder on questions flagged as "homework-style" indicating that "cut-and-paste-ready" answers are not welcome. These doesn't help anyone. Coaching is the correct way to go.

5. Automatically close "homework-style" questions after two weeks of inactivity. They could even be deleted after, let's say, 4 months. That is well enough for people to refer back to the answer they had to a question asked earlier in a trimester. Or for teachers to look around for activity that could be interpreted as cheating.

6. Encourage "homework-style" posters to go around and reply to some other "homework-style" questions. Offer badge for "cooperation". That will reduce the charge of these questions on the rest of the community, help them build stronger knowledge, and help them view the other side of the community aspect of Stack Overflow.

Well, that's it for now. Hope that helps.

Answer 10

Another possible solution might be to bring questions that have more votes to the front of the queue. In other words, if you have 4 close or 2 leave open votes on a question, if it gets brought to the front of the queue, it is more likely to be resolved quickly.

Answer 11

... or, you know, they could just give us more close votes.

It's 9 AM here, and I'm already out of close votes for the day. On most days, fifty votes just isn't enough to deal with new questions plus questions in the Close Vote Queue.

I don't need a prize, or a badge, or even thanks... I just want more freaking votes.

Answer 12

Automatically prioritize posts in the review queue that are tagged with either the users favorite tags or tags selected by the user. This could be limited to e.g. 3 tags if performance is a problem.

If there a no more posts with the specified tags, show other posts (without having to remove the "filter", which is just an annoying UX).

^{Similar suggestion, Feature Request Meta Post}

Answer 13

^{Copied from Reduce the number of reviews needed to complete an item in the CV queue}

---

There are currently 95.3k items in the Close Vote queue, and 2 items in the Reopen Votes queue.

The number of reviews needed to close a question from the CV queue is 5, which is ridiculously high. Here's a stat:

In the past 90 days, 46,506 close flags have been processed; 45,707 were marked as "helpful" (resulted in at least one close vote) and 34,000 have resulted in questions being closed.

That's 73% accuracy from <3k users flagging; i.e. the people we believe to be least-able to recognise a close-worthly question are already getting 3/4 of them right... do we really need a further 5 >3k users to back them up?

---

Here's my actual feature-request:

Let's find a number n, such that requiring only n reviews to end a review results in the same action being taken as requiring 5 reviews, y% of the time.

If we require only 3 reviews instead of 5, and we end up with the same outcome 95/90% of the time as we would do using 5 reviewers, why aren't we doing that already? ^{Yes, I just made those numbers up. Shog help me.}.

---

For those that are dubious as to whether this would open the floor for questions wrongly getting closed, let me calm you down with this;

• Firstly, don't forget that we have the ability to "Vote not to Close" in the review queue. In the normal question view we don't have this. This is why the bar to close can be lower inside the queue than it is out of it, because people can vote for both sides.

• Secondly, my feature-request is not too different to how we already handle the Suggested Edits queue, and a wrong suggested edit is just as reversible as a incorrectly closed question. Hell, we even have a re-open queue; and it's empty. Let's use it.

If you're still struggling, then how about I knock you clean out by suggesting we add the ability to flag a question for reopening. This would force the question into the reopen queue, and handle the small percentage of questions that incorrectly get closed. It would also allieviate a proportion of the flags moderators have to deal with at the moment. If we're worried about the question-owner spamming the flagging option, only enable it for non-owners.

Answer 14

I get the impression that except perhaps for test questions we're mainly getting recent questions to review. I suggest that once you've handled the current maximum 40 questions in a day (or perhaps earlier) you start processing the from the back of the queue, or just generally give us more old questions to process, and probably a higher limit per day in response.

Answer 15

I have another suggestion to make:

Let vote-to-close flags count as silent close votes and convert them to real ones once they have been validated by a user actually voting to close.

Once a VTC-flag gets raised on a question the question enters the review queue. Now the post needs five votes by five different reviewers to be actually closed. Lets say a question enters the queue with two flags raised. A user reviews the question and votes to close. After this voting the close vote count of the question would actually be three, because the two flags cast have been converted to actual votes.

Consequently spoken: When a question has an active close vote (one that has not aged away) a new VTC-flag cast on that question should automaticalle be converted to a close vote and be validated.

To make a long story short: Once a user with the necessary privileges has decided to cast a close vote on a question, let others (whether they have the closing privilege or not) jump on the train by flagging the post.

Answer 16

Allow a metric other than filtering by tag. Such as searching with a text box or sorting by date or sorting by OP reputation.

It doesn't make sense to constantly be reviewing content that was posted 49 minutes ago and is still being edited.

Answer 17

I have some suggestions and ask for more here:

1. Review queues only contains the post which have already two close votes (and asked in the last seven days, but older post should be considered with one vote), as one close vote can not always be correct (some users cast vote just because they do not get the post).
2. Privileges to close the post directly should be at a level where more users can avail it. (Obviously at the level where community can trust them).
3. Three votes of users with more than 5000 reputation points should be enough to close the post.
4. A pre-filter apply to the review queue for the users on the basis of tags they are most active in (If I am active user of jQuery give me list of it, so I don't need skip).
5. Increase the threshold to review the queue from 40 to 60 (or somewhat more suitable)
6. Highlight the close link for more visibility so more users can cast close vote directly from a post.
7. Anti-close votes like Leave Open should be on the post itself, gets visible if a close vote is cast, so a wrong misleading vote can be ignored directly from the post without being reviewed in a queue.
8. One reputation point over 10 reviews (or 10 points over 100 review) will motivate more users to review.

Answer 18

Here are my thoughts (posted under another similar question):

1. In the close queue, show questions based on the reviewer's expertise. (Yes we have the manual filter but it has limitations, such as only allowing 3 tags max.)
2. Make the close queue more transparent.

Answer 19

And some more suggestions:

• Allow more people to review close votes. 18k users have > 3k reputation (Nov 2013), 33k users have reputation score between 1k and 3k. I would suggest to allow users with 1k reputation on SO and 10k global reputation to review the close votes.

• Allow people who already have 1k close vote reviews (and have earned the gold badge) to review more questions per day. These users can't earn another badge so they wouldn't try to cheat the review system.

Answer 20

A suggestion:

Age all posts out of the Close queue after two weeks. The rationale is that if five close votes cannot be accumulated against a post in two weeks, closing the post is likely a dubious decision anyways. Additionally, the close votes against those posts would also be cleared, enabling users if necessary to attempt a reclose of the question, without having to wait for an edit.

Optionally: As a corollary, one might argue that the same criteria be applied to the re-open queue: age out all posts two weeks after an initial re-open vote. However this queue seems to be unmanageable, so symmetry may be unnecessary.

This would not interfere in any way with the ability of the community to rapidly close (and if necessary delete) posts that absolutely must be dealt with quickly. In fact, this proposal might assist in having those posts stand out, enabling them to be dealt with more efficiently.

The immediate effect would be to clear a large number of posts from the Close queue, and with the aging mechanism in place, hopefully allowing it to be maintained at a reasonable size. As mentioned by others, the advantage of seeing a manageable Close queue is increased community interest in maintaining it at a manageable size, presumably allowing more poor quality posts to be closed, more promptly.

If this were to be undertaken on a trial basis, what is the worst that would happen? 74,000 or so pots (less those nominated for closing within the past two weeks) would be cleared. Most of those posts weren't going to be reviewed in less than a few months anyways.

If this change is adopted and becomes successful successful, one could imagine creating a separate long term Disposal-Review queue with the aim of slowly working through and assessing old posts. If they were sorted inversely on a quality measurement, the likely worst offenders (or at least, best candidates for action) could be dealt with soonest.

update: The underlying assumption behind this proposal is that the Close Queue is broken, and needs fixing; that a mechanism which worked well for several years has now been overwhelmed by the site's popularity, and needs refreshing.

Answer 21

From my answer here:

I think a contest is the right approach, but it needs to be done in a big way.

First, set an amount of time, I was thinking two months, though 45 days is fine. If you do that starting next month, you can end things at the end of the year, but that's not crucial.

Second, you give out one-time permanent badges for the contest commemorating their achievement forever. Let's say for 250, 500 and 1000 closes in that timeframe (obviously you should probably remove the review limit for the duration of the contest).

Third, you give out either rep or even specialer one-time ranked permanent badges for the top 3 closers.

Make it as big as the Winter Bash Hat thing, go wild.

Answer 22

Would it not be possible to change the minimum amount of required rep to 2k instead of 3k? I think 2k is already quite hard to get, and proves that you know your way around SO and should be capable of deciding if a question needs to be closed.

More reviewers obviously helps trim down the queue.

Answer 23

The problem still persists - currently the queue has 85,200 questions in it. I like this answer by PearsonArtPhoto. I would add another suggestion to it:

Should we make the minimum reputation level higher for asking questions? This would significantly lower the amount of bad questions that are flooding the queue.

Most of the bad questions are not a result of users being stupid, just lazy. If they would be forced to participate in the community for some time, before being allowed to ask a question it would result in two things:

1. They would ask a question only if they really need an answer. If the amount of time needed to earn, let's say 5 or 10 (or 2 or 20 - just examples) reputation was worth it for the user, then probably this question is not trivial.

2. They would learn more about this community just by participating it, not by reading the help center (let's say this clearly - almost NO ONE reads the FAQ before posting their first question).

Update:

I've made a little research to find out how many "good" questions would be lost, if there was a bar. The procedure was as follows:

1. I've queried SO database to retrieve all closed questions since 2013-01-01 till 2013-10-20 and all opened and up-voted (with score at least 1) for the same time range

2. I've filtered the closed questions to get "questions from fresh users". A "question from fresh user" is a question with creation date distant no more than 5 days from user's account creation date.

3. I've accumulated all user IDs from the previous step, to produce a list of "users who asked a question within 5 days from their account creation which was closed"

4. I counted the "good" questions created by these users. "Good question" is a question that is not closed, and has score at least 1.

The results are:

1. There were 95007 closed questions and 600270 opened, up-voted ("good") questions.

2. 22514 of these questions were "fresh" (23%).

3. These "fresh" questions were asked by 21016 users.

4. These 21016 users asked also 13652 "good" questions. Overall number of "good" and "fresh" questions is 90092.

Out of 600270 "good" questions 13652 is around 2.3%, and total percentage of "good" questions from "fresh" users is 15%. Therefore, it seems if for given time range a bar was introduced probably(*) there would be 23% less questions to close, and 15% good questions lost.

If we define "good" question as a question with score 3 or more, then only 2.1% would probably (*) be lost (12637 all opened questions with score at least 3).

If it would be possible to filter out users who asked a "bad" question not using a global rep bar (for example: introduce rep bar after first asked question was downvoted heavily or closed) than only 0.3% of "good" questions would've been lost (1986 questions with score at least 3 from users with closed "fresh" questions).

(*) it's "probably" because it is still possible that these questions would have been asked anyway, after the user would reach the bar

Explanations:

• why the time range?

  I started downloading the data on 2013-10-20, and it took me a while.

• why check day-distance between asked question and account creation time, and not reputation itself?

  It is practically impossible to retrieve user's reputation value for a distant past.

• What is the query for closed questions?

https://data.stackexchange.com/stackoverflow/query/152649/list-of-closed-quesions-from-2013

(note the throttling - you may need to run the query more than once, uncommenting the id condition with appropriate id each time)

• What is the query for opened up-voted questions?

https://data.stackexchange.com/stackoverflow/query/143859/open-upvoted-questions-2013

(note the throttling - you may need to run the query more than once, uncommenting the id condition with appropriate id each time)

Answer 24

At time of writing, there are 120K+ close vote questions.

Seeing as something drastic needs to be done, there need to be more harsh actions taken, which could looks as follows:

1. Automatically close/delete questions that receive a certain negative score (say -3 or -5). This reduces the time/workflow required to close vote poor questions.

2. Give some kind of reward when downvoted questions are eventually closed. It should be in the form of reputation, because if a number of people agree together that a question is bad, they have helped clean up the site by removing it. Maybe give each user that helped close the question by downvoting it +1 or something.

By way of example, for point 2, if you helped close 20 questions by downvoting it, you get 20 extra reputation, which is not exhorbitant. It's just enough to say thanks, but not so much that people will start closing questions that are actually valuable.

You should perhaps then increase the reputation required to get the downvoting privilege, because to someone with 125, downvoting will look like an easy way to gain reputation.

Sure, you don't get all your stats about why stuff was closed, but things are too desperate now to worry about that.

Answer 25

To reduce the backlog:

Gamify closing more.

Right now people review the close votes queue to get badges and for the love of the site. If that's not sufficient to clear the queue, add more incentives. SE is an excellent example of how well gamification can drive participation in an otherwise not very exciting activity. Why not apply the same principle here?

Obviously this will require some careful thinking to avoid incentivizing undesirable behavior, but that's the sort of tinkering that SE requires generally so it's nothing new.

---

One way to gamify the close queue without incentivizing bad behavior. This is meant more as a proof of concept than a completely foolproof system, something to distinguish between incentivizing well and incentivizing poorly.

Each close vote cast earns 1 rep for the caster if the question is subsequently closed. Each Do Not Close vote earns 1 rep for the caster if the question reaches five DNC votes before reaching five close votes (i.e., being closed). If a caster is on the wrong side of the majority, voting DNC on a question that everyone else votes to close, the caster loses a point. Reviewers aren't shown others' votes to avoid blindly voting with the majority.

In this system, there's a reward for reviewing questions that should potentially be closed, but only if other users agree with you. This is meant to be exactly analogous to the general SE model in which questions and answers are rewarded if and only if the community judges them useful.