New Feature: Community Review Tasks - Now in Beta

Question

Please post new bug reports, feature requests or discussions as individual questions tagged with at least review in addition to the requisite

And thanks for all the great feedback so far!

---

We are currently soliciting feedback for an improved version of /review to help with the very important task of maintaining site quality. There are three targeted tasks we have built thus far and will continue to roll out others after we gain confidence these three are working:

• Low Quality Posts: This task targets posts that we've algorithmically determined to be low quality. Your job is to read these posts and:

  • decide if a post does in fact Look Good
  • Edit the post to make it better
  • Delete¹ the post entirely
  • or skip it if you are Not Sure.

  The minimum reputation to perform this task corresponds with the ability to edit any post (currently 2,000).

• Suggested Edits: This task targets posts edited by users who have not yet earned the reputation to fully edit. Your job is to read these edits and:

  • decide to Approve the edit as is
  • Reject the edit
  • Edit the original post and make your own edit
  • Delete the post (moderator only)
  • or skip it if you are Not Sure.

  The minimum reputation to perform this task corresponds with the ability to edit any post (currently 2,000).

• Close Votes: This task targets posts that have received at least one close vote. These posts generally need additional review to help decide if a post does in fact require closure. Your job is to read these posts and:

  • decide to Close this post by casting another close vote
  • Do Not Close² this post if do not agree with the reasons for closing
  • Edit this post to make it better
  • or skip it if you are Not Sure.

  The minimum reputation to perform this task corresponds with the ability to cast close votes (currently 3,000).

Each of the actions on the review task listed above may not always be available based on your current reputation or moderator status. We will be reviewing the data collected in this experiment closely to determine if these new review tasks are working and also reserve the right to pull these tasks out entirely if we find they are not. We have enabled these review tasks on Meta, Super User, Server Fault, and Stack Overflow.

To check out the new hotness visit: https://meta.stackoverflow.com/review

^{1 If you don't have enough rep to delete, you can "Recommend Deletion". This doesn't do anything right now except move it to the front of the queue for people who can delete it.
2 "Do Not Close" just removes it from the queue right now -- it doesn't affect close votes.}

Answer 1

So I've been reviewing low quality posts on SO. There are four options:

• Looks Good
• Edit
• Recommend Deletion
• Not Sure

What about if I want to recommend an answer be turned into a comment? I've seen many so-called answers which would be useful as comments instead of just being deleted.

Answer 2

Not sure if anyone mentioned this, but "Low Quality Posts" could be further segmented into

• "Low Quality Questions"
• "Low Quality Answers".

I had to stare hard at an "answer" before I realized it was actually a question.

It would make reviewing faster because you don't have to switch sentiments whenever the UI switches between showing you a question and showing you an answer.

Just sayin'.

Answer 3

Voting Up or Down is Missing

Under the old system, you could vote posts up or down while reviewing them. Since voting is one of the metrics tracked for the Reviewer badge, why is this ability not present in the new review interface?

Answer 4

I liked the bits where it showed progress towards badges i.e.

1. Copy Editor
2. Electorate

I find stuff like that quite motivating (sadly). I notice they are now missing. Can they be reincluded or put somewhere else.

Apologies if I've missed them

Answer 5

Few comments:

• Can the confirmation for delete please be removed? If not, provide an option. It's the /review route. I know a lot will be up for deletion. Having a confirmation is really annoying. At least skip the confirmation for Moderators
• It would be great if all answers were visible, below the question. Having that helps to know whether it should be moved to a comment or deleted altogether
• After approving/rejecting a suggested edit, the other button ( approve/reject) still remains. Fixed
• Also, why aren't the voting arrows present on the posts? :/

Answer 6

Users are locked out of reviewing Close Votes when they run out of close votes for the day.

Consequently, the Top Reviewers each day are the users with the greatest "do not close"-vote to "close"-vote ratio, because they are able to accumulate the most reviews before being locked out. These users also have an advantage in the all time Top Reviewers list -- I took an early lead, but lost it and continue to fall further behind because I cast fewer "do not close" votes than certain other users.

This current design will skew votes towards "do not close", or at least encourage users to select "not sure" instead of "close" to avoid getting locked out. Instead, "close", "do not close" and "edit" should all count towards a daily review limit.

(Why am I including "edit"? It's good to encourage edits, but once we include "do not close", having a larger number of "edit"s would be the only way to get an advantage and would be the new factor determining the daily Top Reviewers. This would lead to a lot of frivolous edits bumping questions that should actually be closed.)

After that, consider removing the daily Top Reviewers list. It will be a lot less interesting once everybody's tied at a shared cap.

Answer 7

In the old review-process I could restrict reviews to the tags I understand. In the new review I am supposed to give comments to things I do not understand.

Thus, I would like to have my tags back!

---

For clarification, these work in the old process:

• https://stackoverflow.com/review/late-answers?tag=perl
• https://stackoverflow.com/review/first-questions?tag=perl
• https://stackoverflow.com/review/first-answers?tag=perl

These do not work in the new process:

• https://stackoverflow.com/review-beta/close?tag=perl
• https://stackoverflow.com/review-beta/suggested-edits?tag=perl
• https://stackoverflow.com/review-beta/low-quality-posts?tag=perl

Answer 8

Awesome, glad to see you guys aren't slacking. SO's a great site now and will get even better. I frequent the review tab a lot, but my rep's < 2000, so I can't access any of those tasks. Can you offer the beta only to those who can access it? Currently, I'm seeing this every time I click review.

Answer 9

Some of questions raised due to new review system?

1. How/Does/Where review stats going to be displayed.

2. How new review system will affect the review stats and it's parameters? Why not to show all parameters done in review stats?
   

3. How/where the badge progress on review page(Copy Editor, Electorate) will be shown?

4. If we click on Not Sure in reviewing low quality posts, then will those be counted in reviewed posts.
5. Does First answers and First questions are not required to review according to review system?
6. Why not given close option for low quality posts? Does only answers shown in low quality posts?
7. Why can't we vote on posts according to new system during review?

Answer 10

I ran into a problem with this today. I noticed a poor edit suggestion and went to just improve the post and do it right. I went in, edited the post, submitted it, and I was moved on to the next suggestion. I went to the thread with the actual post (thanks to the browser history) to review the question and I noticed that my edits didn't take effect. I went in from the post itself (rather than the new review queue) and re-made my edits. I was given an error that the title already existed.

Apparently if there is an error making an edit in the queue it just swallows the error, displays nothing to the user, and moves on giving the impression that it worked even if it didn't.

A few unrelated comments:

• I also miss not having the number of approve/reject votes displayed, or the reject reasons displayed, as on the old permanent edit suggestion page. It also doesn't list the approval/suggestion stats or recent approvals/suggestions (which is less useful than reject reasons, but still useful).

• If you hit 'not sure' for an edit and then go back to it it says you've already voted and can't vote again. While I can understand approve/reject votes being locked, and it's nice to not have something I've "not sure-ed" show up again in the queue, I would like to change my vote. This was prompted by a mis-click when I meant to reject a suggested edit.

• When you run out of votes it would be nice of the approve/reject buttons just went away, and there was some indication at the top of the page so that I don't spend time looking through questions that I can't vote on.

Answer 11

Sub-feature-request

Navigating the close menu for dupes is annoying. It AJAX requests every time you proceed to the next stage for dupes, and the other close reasons involve two clicks and a wait. Not that big an issue--but it disrupts the flow of the thing, especially for dupes.

The feature request is as follows:

For all options but dupe: just select the close reason beforehand so we can click "close" and be done with it.

For dupe, open the close menu at the point where you choose which question to make it a duplicate of. Even better, do this:

Enable the "duplicate" buttons only if the user has looked at the respective tabs.

Answer 12

One thing I don't like about the new suggested edit list is that it's no longer a list.

It used to show a list of up to 30 suggested edits, which made it easy to quickly scroll through them looking for things I actually know something about.

The new list shows me just one suggested edit, and I have to explicitly skip it (by clicking "Not Sure") before I can see another. It makes reviewing a lot of suggested edits much more awkward.

An unrelated thought: Should there be an option to show suggested edits tagged with my favorite tags, or to push them to the top of the list? Those are the edits I'm much more likely to be able to judge. Posted separately here.

Answer 13

The Review - close votes section is heavily gimped - I can no longer sort/filter by number of close votes or close reasons.

Please, please get this on the new tasks.

Answer 14

What about the possibility of waiving the 2k reputation requirement for users who have earned the Marshal and/or Reviewer badge? I realize this is a small percentage of users, but I'd hate to see reviewing be taken away from those of us who have put in a lot of time on the old reviewing system, and who are still working to achieve the 2k goal.

Answer 15

We can add comments inline now, why can't we have inline:

• Voting
• Editing
• Flagging
• 10k/mod "edit tags"-ing

Answer 16

May "Not Sure" be replaced with "Next"?

If I want to pass to the next post is not only because I am not sure, but also, e.g., because I am reviewing low quality posts, and I am shown a question that I would vote to close, but I cannot because I have already voted to close all the questions I could.

Answer 17

Proposed Edits on Accepted Answers

It should be much more obvious (color coding? different border? clown cars and sirens?) when a proposed edit modifies an accepted answer. The tiny "accepted yes" on the right-hand side doesn't really leap out; you have to make a conscious decision to check for it each time.

Answer 18

There are actions I want to take on some posts but can't within the review interface, such as flagging or adding a comment (though this is allowed for low-quality posts), and I end up having to open the post separately to do so. Can you add more choices so that this isn't necessary? For example, old questions that should be closed but may not get enough close votes or should be migrated to a site not listed in the close dialog may be better handled by adding a moderator flag.

Answer 19

In a lot of cases I get an answer that's recommended for deletion, but often the question (and any other answers are crappy too). At the moment we lack a lot of context, because we can only see one answer and the question it was given on.

So I'd like an easier way to simply delete or close the question too, without having to open it on a new tab, delete the question, then go back and delete the answer (even though its already deleted).

It might be useful to add a check to see if any other posts on the same question are somewhere in the queue and if so, add them to the screen like the question is at the bottom.

Answer 20

This is both fantastic and marvelous.

Can we have keyboard shortcuts for taking action? It would be lovely.

Say a for accept, r for reject, n for not sure, i for improve.

Answer 21

(I had posted this as an addendum to another answer, but I thought it should be separate.)

I'd like to have an option to show suggested edits tagged with my favorite tags, or to push them to the top of the list (or, if there is no list, to give them some preference). Those are the edits I'm much more likely to be able to judge.

Answer 22

Under "Review Low Quality Posts" (under /review-beta), when reviewing an answer, the question is showed below the low quality answer. I would like to also have the other, competing answers listed.

If I find a low quality answer which is downvoted I'm much more inclined to vote for deletion if I know that there are other, high quaility answers (perhaps even one marked as accepted).

If the low quality answer is the only one, it's worth spending time on improving it, but if there's already an excellent upvoted, accepted answer in there it's just a waste of time.

Answer 23

The interface lagged while I was trying to submit an edit from the Close Votes review page. The submit button appeared to be disabled after the first submission, but I think I pressed enter multiple times.

It looks like the client let me send the same edit multiple times, and the server accepted them all without collapsing it to a single edit:

Answer 24

In the Close Votes queue, I would really appreciate the ability to see the answers that have been posted to a question without having to click through to the question. A link ("View Answers") that would unfold the answers below the question would be ideal.

Like others here, I don't always have total confidence in my ability to identify duplicate questions about unfamiliar topics. On Stack Overflow, sometimes a question will concern an error that could have several different explanations, and one needs a pretty detailed knowledge of the topic in question to tell which explanation is correct and solves the problem. In that case, two questions could look like duplicates, but actually have completely different solutions, and a close vote would not be appropriate.

However, if the two questions have also generated very similar answers, and there's some indication that those answers were correct, then the chance of erroneous closure is lower and a close vote is warranted.

As an aside, I'd second other calls for the ability to filter by (at a minimum) tag and close vote type. With that in place, I think this will be a vastly more useful interface for reviewing questions with close votes/flags than the old flag interface. Thank you all for working on this!

Answer 25

This new system has many advantages over the old one, I believe that the only thing I really (but really!) miss is the ability to sort by tag, and some sort of relevance.

A couple of observations, though:

In the low-quality posts section, when presented with a short answer, the problem may not so much be the answer, but the question. It would be useful to have a simple way to work on the question, for instance voting to close it.

I was presented with something like 75% (if not more) of really old answers to equally old questions, where most of them already had accepted answers. Do we really have to wade through tens of thousands of old answers that the community has already relegated to the bottom of the pages by not voting for them? I am all for cleaning up stuff, but this is a bit too much even for me.

A suggestion could be to order the answers presented for review so answers with no votes and for which there is another, accepted answer are presented last in the queue. Especially if there has been no activity on the question for several months.

Similarly, in the close votes section, I was presented with a long series of questions on subjects that I didn't really feel an expert in (the lack of tag filtering again) and which were quite old (nearly a year). Feels a bit strange to be asked to spend time and effort on so many old and forgotten (many didn't show any activity for the last 10 months or so) questions where I don't really feel I have the insight to, for instance, remove a close vote.

Being able to up- and down vote these old questions would be a nice compromise. I'd feel more at ease just voting for a question in a field where I don't feel expert enough to decide whether it should be closed or not.

Lastly, the wording of the close reasons is slightly confusing, if not downright off-putting. Often a question is just a bad question according to the site's standards and expectations (that questions should show some self effort, be researched, etc.). In several occasions "Not a real question" is not correct - they are real questions, just not good ones.

In the case of a question that has lingered for months, asked by a user with a rep in the single digits, even if one leaves a polite comment, chances are that nothing will happen. There should be a simple and accepted way to close these questions for what they are, not for some bogus reason (too localized, narq, etc.).

Answer 26

If you don't have enough rep, the page just kind of...doesn't work. It's not immediately clear why I can't perform any of these tasks.

As a 500 Stack Overflow rep user, I see this:

Ideally, there could at least be something explaining the requirements (and that I don't meet them) for each task I don't qualify for.

So apparently it does say you need at least 2k reputation to review suggested edits. Maybe make that text red (when I don't have enough rep) to draw attention. I literally did not see that when looking over the page.

Answer 27

Found a slight quirk.

When a question has a close vote on it, the duplicate is already closed and you vote to close as a duplicate as is suggested by /review you get this error:

As is the case with this question the most likely offender is going to be in the form of migrated cross-site duplicates getting closed out and people vote to close different questions as a duplicate with one "winning" the close and the other being left with close votes.

Do these type of duplicate questions need to be hidden from /review?

Answer 28

Under the "Close vote" section, it would be nice to have a smaller "reason selection" dialog pop up closer to the top of the screen instead of having the normal VtC dialog appear when Close is selected. Having to respond to the normal one interrupts the workflow.

It might be nice to do away with the normal VtC dialog all together when reviewers were simply assenting to the already selected close reason, but not making it so convenient that it was possible to accidentally click through.

Answer 29

Current behavior:

http://pasteshack.net/images/517641001342191535.png

Reproduction:

1. Run out of delete votes.

2. Try to delete yet another post.

3. Get the above error, trying to delete from the question page itself and note I ran out of votes.

4. Be forced to press Not Sure instead to get rid of it showing up again and again... >_<

Expected bahavior:

• Low Quality Posts: Replacing "Delete" by "Recommend to Delete" when I ran out of close votes.

• Close Queue: Replacing "Close" by either a flag or an appropriate message to stop the user.

Answer 30

I like this so far. However, while looking at the Low quality posts section, I had issues with it. The only options to choose from are Looks good, Edit, Recommend deletion, and Not sure.

What does Recommend Deletion do? Does it make a flag or something? I wouldn't mind having the flagging option somewhere on this page, I feel it's sometimes necessary. I may want to flag it as "Not an answer" sometimes, or point out advertisement.