Short version:

  1. Make a filter selection page the default entry point for the review queues
  2. Show the size of the queues for tags they are likely able to work with - not the whole queue
  3. When all review options are done, show the user the impact they have had on the view of the queue they were given to start with

Long version:

There are (at least) two things that discourage people from getting into the /review - the appearance of an endless list that one can't make a dent on and having to 'skip' all the time to get to questions that one does know how.

I'll also point out that I suspect that some honest review audit failures come from people trying to review things they don't have a grasp of. Making it easier for people to review what they do know may make it easier to honestly pass audits.

Under this model instead of showing someone trying to make a dent in "84,803 need review", break it down to the tags and reasons that a person can work with and see a reward from.

I'm going to use Robert Harvey for this because he has more interesting tags than I do. I'm also going to pretend that everything he answered or asked is in the close review queue for the purposes of data.

Take the top 50 tags a person has. This is about the first page of tags. The reason for this is that it helps ensure that there is enough to build up a queue. It also helps identify the questions that a person is likely able to cast the appropriate close vote (or leave open) on without guessing about technologies one is less familiar with.

Instead of clicking on the review and just getting at the queue with that little filter link at the top, one instead is taken to a filter selection page where one can build up the appropriate filter to work from.

You could select wedges from the pie chart, or select them from the list on the side. Furthermore, there would also be a place to enter other tags.

Additionally, there would be a breakdown by close reason that one can select to build up the appropriate filter.

With both tags and close reasons, a question can fall into multiple spots. For the purposes of this, show it in both tags or reasons.

Upon completing all the reviews for the day, show the information again, and allow them user to see the impact on the tags and reasons they chose. This is the reward - not a badge or something, but a visible impact on the size of their corner of the queue.

One of the things this may help do - by presenting this information - is hep engage the various tag communities around the site to go and work on the review queue for their tag. Seeing that some tag went from 3894 questions in the queue to 3563 in the last day may help.

Allowing a person to go and address just the off topic - recommend or off topic - migration (note, those are time sensitive) -- or even just 'off topic').

Of course there should still be a 'just take me to the queue' (and one can still set up the filter in the queue that way). The key is to show the impact of what they have done to the queue in a way that is meaningful and not overwhelming.

A mockup to give an idea of what I'm thinking about. Don't look too deeply into the numbers, they are more as an example.

You hit your standard 'review' link to get to the close review queue. Instead of taking you right to the reviews, you get a page of stats that lets you see what is in the queue and modify the filter.

first click after the review link

Note the 'Other' box at the bottom which has the same functionality as the 'tag' input that one can get from the current filter link.

I should also point out that the 'Review!' link can be clicked on without setting up any filters if you want to just "take me to the queue" and tackle whatever is there or use the old filter style setup.

You go through, you do your reviews and then you have a page that shows the queue once you've exhausted your tag or your reviews for the day.

after reviewing

Thats really what its about. Make it easy to set up the filter of things you are likely able to review, make it easy to see what effect you have had on the queue when you are complete with the review.

  • 12
    this would lead to me spend more time in the close queue for sure Commented Oct 19, 2013 at 22:26
  • [asp.net-mvc] is on there twice.
    – Cole Tobin
    Commented Oct 20, 2013 at 3:37
  • 1
    @ColeJohnson I clearly need to practice my copy and paste skills more.
    – user213963
    Commented Oct 20, 2013 at 3:41
  • 1
    Systemically speaking, this is beautiful!
    – brasofilo
    Commented Oct 20, 2013 at 5:11
  • 2
    @brasofilo I think you mean statistically speaking ;)
    – Cole Tobin
    Commented Oct 20, 2013 at 5:42
  • 2
    Great idea, but 3 comments. (A) Save the pies for dessert.. Arranging with bar charts would be much nicer. (B) Arrange the close reasons by the order they appear in the close vote list. (C) allow filtering by nothing, by close reason, or by tag (or both) -- don't force people to click through (a handy menu on the side showing the last 10 review filters you made and what the number of pending items are would be nice)
    – jmac
    Commented Nov 21, 2013 at 23:42
  • 2
    @jmac In my defense, I was more going for a 'this is an idea...' not 'this is a display that it should be'. A tree map might be appropriate if one could visualize the tags in it, somehow... but then thats why I don't do UI. The key point is we need a way to both rapidly select filters and see the impact we had on the queue for the filters. If this is taken seriously (chicken & egg?) I would certainly put much more thought into the design (and maybe even a question or two on UX.SE).
    – user213963
    Commented Nov 22, 2013 at 1:45
  • how do you expect this to work for small CV queues? Or this idea is targeted primarily at stackoverflow?
    – gnat
    Commented Nov 22, 2013 at 13:55

4 Answers 4


Update Jan 3rd, 2017: As noted below, this is implemented - however, it cannot be enabled for tags on Stack Overflow, as it will bring the site to its knees.

Close reason filters from /review are still operational, and tags as well on smaller sites (currently Super User and Ask Ubuntu).

This feature was implemented and went out in build rev 2014.5.16.2252 on meta and 2014.5.16.1610 on sites.

behold the freehand circle

works on tag pages as well

Currently it's enabled only on Stack Overflow. Note, that we have a big threshold (50) for how many outstanding reviews there must be with a tag or close reason in order for it to show up on the dashboard. Also, as opposed to the mockup, we don't display counts, since those are very expensive if we'd want exact numbers for each user.

As of build rev 2014.5.19.1613 we don't show any of your ignored tags on the review dashboard.

  • When I use one of the links I cannot change the filter of the Close View. To be more specific, when clicking on duplicate and the adding the c#-Tag as filter the c#-Tag is not applied to the query. Is this by design?
    – Jehof
    Commented May 19, 2014 at 9:14
  • 2
    @Jehof yes, that is a known bug
    – m0sa
    Commented May 19, 2014 at 9:18
  • Is there any possibility of personalizing the tags displayed, or is that also too expensive?
    – mhlester
    Commented May 19, 2014 at 15:22
  • 2
    @mhlester hmn... we could hide your ignored tags..
    – m0sa
    Commented May 19, 2014 at 15:25
  • 2
    that's not a bad idea. maybe also bump favorite tags? i've never found the need to ignore tags
    – mhlester
    Commented May 19, 2014 at 15:27
  • @mhlester we discussed bumping favourite tags and decided that it was not a good idea since they may not be enough outstanding reviews. the are tags displayed are ranked on the global outstanding review count (not per user) base so you can quickly run into the scenario where the user has no more reviews for in a tag but would still see the link...
    – m0sa
    Commented May 19, 2014 at 15:33
  • I just saw @Shog's revision notes (couldn't pick it out on its own). That does an even better job than i'd hoped for!
    – mhlester
    Commented May 19, 2014 at 15:37
  • What exactly is enabled only on SO and what is enabled everywhere?
    – Braiam
    Commented May 19, 2014 at 23:59
  • So you're testing the link validation queue again?
    – hichris123
    Commented May 28, 2014 at 10:49
  • it's (still) visible to devs but we're not working on it
    – m0sa
    Commented May 28, 2014 at 10:55

I like this idea, but I hate the notion of a click-through page, and I really hate those big honkin' pie charts.

David Fullerton proposed a fairly simple UI change that could make filters a good deal more obvious without unduly disrupting the UI:

list common filters and counts below top-level review item

Keeping those counts up-to-date is expensive, so we'd probably have to limit them to the broadest possible filters; still, this could go a long way toward introducing folks to the idea of filtering, as well as providing one-click access to each specific close reason.

It would be nice to provide our more active reviewers who have saved filters with a count as well, but I'm not sure this is feasible.

Finally, I would like to bring this whole concept out of /review and drop it directly onto the individual tag pages. Imagine visiting and seeing a little red indicator near the top calling out the number of pending review tasks for that tag... Clicking takes you directly into review.

bat signal

I'm most excited about the possible effects of the latter; in fact, I'd like to start testing it immediately.

  • 2
    There's a very good reason why I tend to stay on the backend near the databases and such... keeps people from getting pie on their web pages.
    – user213963
    Commented Dec 3, 2013 at 20:38
  • do I understand it correctly that reviewers with default filter settings will see this as well (ie will learn that there are filters and how filters matter)?
    – gnat
    Commented Dec 4, 2013 at 7:04
  • 1
    That's the idea, @gnat.
    – Shog9
    Commented Dec 4, 2013 at 16:11

I'm enamored with the idea, but:

  • You've increased the number of clicks I have to do to get to a review queue from 2 to (up to) 4
  • The global number would still persist, or we're all just lying to ourselves (even if we're looking at only the C# chunk, there's still better than 80,000 questions left to review)
  • I'm not sure what your solution accomplishes that the filters already don't, besides providing more statistics about the queue

Actually, maybe that's what sets this suggestion apart - statistics. I am in firm agreement with showing the statistics of what questions need to be reviewed.

  • 1
    Only 1 additional click to get to the queue (I see I need to mock it out a bit more) - "Of course there should still be a 'just take me to the queue' (and one can still set up the filter in the queue that way)." The key is to get people into doing the reviews. Many times I've seen comments to the effect of "you mean, you can filter the queue to just see off topic?" - they aren't the most obvious things to find or set up. This makes them, well, in your face at the start - to create and set up and see the effect upon when complete.
    – user213963
    Commented Oct 20, 2013 at 10:05

Reviews are not really tag specific (mostly)

While I've not ventured in the close vote queue, with my meagre 800rep, I imagine it doesn't matter what the tag is most of the time.

I've reviewed things in tags plenty where it doesn't matter what the tags are.

I hate everything and so am posting this funkyness cos I'm blur brained - wibble wibble I am a fish

I can easily review that, even if it was in the "rocket-science" tag... and I imagine similar scenarios are most common for the close vote queue too. ie it's clear cut.

There's a potential that people will go to their favourite tag only.
Currently users are reviewing everything thrown at them, and so currently might do a few in certain tags which given the choice of specific review tags they would not.

If the same people are willing to do the same amount of work, but on specific tags only, then when those tag reviews are empty (for the smaller tags) they do nothing, whereas before they might have reviewed another tag.

Will people do more just because it's a tag they like? Does it really matter, as they're not answering questions on their favoured tag.

The flip side, is it might actually work better in terms of people doing more reviews than they currently do now "because" they can choose tags of their interest.
So the tags with the biggest percentage of close votes might actually get more done.

Obviously the main issue with the close vote queue is it's a never ending queue so people get bored. They do what they can, leave.
If they could do a load on their favourite tag, they might do more than they would have.

Especially as the queue would be seen to be deteriorating. In fact, it could become a bit of a competition between tags as people try to get theirs down faster (I hate this thought, like being back at nursery, but it does happen/work)

And to contradict myself from the against [gracefully: as there is a valid point]:
If the same people are willing to do the same amount of work, but on specific tags only, some tags might get more action and thus people actually do more work as they enjoy reviewing their tag of choice more than just any thrown at them.

Although, I'm not convinced this would be the case on close vote queue, as even people loving certain tags might not make them more likely to do more close votes.


I'd say it'd be well worth trying, but it's likely a s*#t load of dev work for what is a case of testing the water. If you could give more conclusive reasoning as to why this would work, stronger theories, perhaps people saying here if they'd do more if it was a tag of their choice, it might be considered*

(*no, it wont... )

Everyone: If you would do more on the close queue if you could do chosen tags, upvote Kate Gregory's comment!

  • 1
    While there are definitely many questions that are obvious to see that they should be closed, there are also plenty that aren't obvious - many requiring domain knowledge to analyse properly. Commented Oct 20, 2013 at 2:42
  • 4
    I take umbrage to your phrase "I imagine it doesn't matter what the tag is most of the time". Context does matter, especially to tell the difference between a question that could use a little love, or one that's been asked for the umpteenth time.
    – Makoto
    Commented Oct 20, 2013 at 3:46
  • Well, my main point is, implement this and people will stop doing all tags except theirs, so don't expect help on any others. It could be detrimental to the queues.
    – James
    Commented Oct 20, 2013 at 12:50

You must log in to answer this question.