Easy close review queue filtering interface

Question

Short version:

1. Make a filter selection page the default entry point for the review queues
2. Show the size of the queues for tags they are likely able to work with - not the whole queue
3. When all review options are done, show the user the impact they have had on the view of the queue they were given to start with

---

Long version:

There are (at least) two things that discourage people from getting into the /review - the appearance of an endless list that one can't make a dent on and having to 'skip' all the time to get to questions that one does know how.

^{I'll also point out that I suspect that some honest review audit failures come from people trying to review things they don't have a grasp of. Making it easier for people to review what they do know may make it easier to honestly pass audits.}

Under this model instead of showing someone trying to make a dent in "84,803 need review", break it down to the tags and reasons that a person can work with and see a reward from.

I'm going to use Robert Harvey for this because he has more interesting tags than I do. I'm also going to pretend that everything he answered or asked is in the close review queue for the purposes of data.

Take the top 50 tags a person has. This is about the first page of tags. The reason for this is that it helps ensure that there is enough to build up a queue. It also helps identify the questions that a person is likely able to cast the appropriate close vote (or leave open) on without guessing about technologies one is less familiar with.

Instead of clicking on the review and just getting at the queue with that little filter link at the top, one instead is taken to a filter selection page where one can build up the appropriate filter to work from.

You could select wedges from the pie chart, or select them from the list on the side. Furthermore, there would also be a place to enter other tags.

Additionally, there would be a breakdown by close reason that one can select to build up the appropriate filter.

With both tags and close reasons, a question can fall into multiple spots. For the purposes of this, show it in both tags or reasons.

Upon completing all the reviews for the day, show the information again, and allow them user to see the impact on the tags and reasons they chose. This is the reward - not a badge or something, but a visible impact on the size of their corner of the queue.

One of the things this may help do - by presenting this information - is hep engage the various tag communities around the site to go and work on the review queue for their tag. Seeing that some tag went from 3894 questions in the queue to 3563 in the last day may help.

Allowing a person to go and address just the off topic - recommend or off topic - migration (note, those are time sensitive) -- or even just 'off topic').

Of course there should still be a 'just take me to the queue' (and one can still set up the filter in the queue that way). The key is to show the impact of what they have done to the queue in a way that is meaningful and not overwhelming.

---

A mockup to give an idea of what I'm thinking about. Don't look too deeply into the numbers, they are more as an example.

You hit your standard 'review' link to get to the close review queue. Instead of taking you right to the reviews, you get a page of stats that lets you see what is in the queue and modify the filter.

Note the 'Other' box at the bottom which has the same functionality as the 'tag' input that one can get from the current filter link.

I should also point out that the 'Review!' link can be clicked on without setting up any filters if you want to just "take me to the queue" and tackle whatever is there or use the old filter style setup.

You go through, you do your reviews and then you have a page that shows the queue once you've exhausted your tag or your reviews for the day.

Thats really what its about. Make it easy to set up the filter of things you are likely able to review, make it easy to see what effect you have had on the queue when you are complete with the review.

Answer 1

Update Jan 3rd, 2017: As noted below, this is implemented - however, it cannot be enabled for tags on Stack Overflow, as it will bring the site to its knees.

Close reason filters from /review are still operational, and tags as well on smaller sites (currently Super User and Ask Ubuntu).

---

This feature was implemented and went out in build rev 2014.5.16.2252 on meta and 2014.5.16.1610 on sites.

Currently it's enabled only on Stack Overflow. Note, that we have a big threshold (50) for how many outstanding reviews there must be with a tag or close reason in order for it to show up on the dashboard. Also, as opposed to the mockup, we don't display counts, since those are very expensive if we'd want exact numbers for each user.

As of build rev 2014.5.19.1613 we don't show any of your ignored tags on the review dashboard.

Answer 2

I like this idea, but I hate the notion of a click-through page, and I really hate those big honkin' pie charts.

David Fullerton proposed a fairly simple UI change that could make filters a good deal more obvious without unduly disrupting the UI:

Keeping those counts up-to-date is expensive, so we'd probably have to limit them to the broadest possible filters; still, this could go a long way toward introducing folks to the idea of filtering, as well as providing one-click access to each specific close reason.

It would be nice to provide our more active reviewers who have saved filters with a count as well, but I'm not sure this is feasible.

Finally, I would like to bring this whole concept out of /review and drop it directly onto the individual tag pages. Imagine visiting java and seeing a little red indicator near the top calling out the number of pending review tasks for that tag... Clicking takes you directly into review.

I'm most excited about the possible effects of the latter; in fact, I'd like to start testing it immediately.

Answer 3

I'm enamored with the idea, but:

• You've increased the number of clicks I have to do to get to a review queue from 2 to (up to) 4
• The global number would still persist, or we're all just lying to ourselves (even if we're looking at only the C# chunk, there's still better than 80,000 questions left to review)
• I'm not sure what your solution accomplishes that the filters already don't, besides providing more statistics about the queue

Actually, maybe that's what sets this suggestion apart - statistics. I am in firm agreement with showing the statistics of what questions need to be reviewed.

Answer 4

Reviews are not really tag specific (mostly)

While I've not ventured in the close vote queue, with my meagre 800rep, I imagine it doesn't matter what the tag is most of the time.

I've reviewed things in tags plenty where it doesn't matter what the tags are.

I hate everything and so am posting this funkyness cos I'm blur brained - wibble wibble I am a fish

I can easily review that, even if it was in the "rocket-science" tag... and I imagine similar scenarios are most common for the close vote queue too. ie it's clear cut.

Against
There's a potential that people will go to their favourite tag only.
Currently users are reviewing everything thrown at them, and so currently might do a few in certain tags which given the choice of specific review tags they would not.

If the same people are willing to do the same amount of work, but on specific tags only, then when those tag reviews are empty (for the smaller tags) they do nothing, whereas before they might have reviewed another tag.

Will people do more just because it's a tag they like? Does it really matter, as they're not answering questions on their favoured tag.

For
The flip side, is it might actually work better in terms of people doing more reviews than they currently do now "because" they can choose tags of their interest.
So the tags with the biggest percentage of close votes might actually get more done.

Obviously the main issue with the close vote queue is it's a never ending queue so people get bored. They do what they can, leave.
If they could do a load on their favourite tag, they might do more than they would have.

Especially as the queue would be seen to be deteriorating. In fact, it could become a bit of a competition between tags as people try to get theirs down faster (I hate this thought, like being back at nursery, but it does happen/work)

And to contradict myself from the against [gracefully: as there is a valid point]:
If the same people are willing to do the same amount of work, but on specific tags only, some tags might get more action and thus people actually do more work as they enjoy reviewing their tag of choice more than just any thrown at them.

Although, I'm not convinced this would be the case on close vote queue, as even people loving certain tags might not make them more likely to do more close votes.

Conclusion

I'd say it'd be well worth trying, but it's likely a s*#t load of dev work for what is a case of testing the water. If you could give more conclusive reasoning as to why this would work, stronger theories, perhaps people saying here if they'd do more if it was a tag of their choice, it might be considered*

(*no, it wont... )

---

Everyone: If you would do more on the close queue if you could do chosen tags, upvote Kate Gregory's comment!