What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 134 Stack Exchange communities.

Please post new bug reports, feature requests or discussions as individual questions tagged with at least in addition to the requisite

And thanks for all the great feedback so far!


We are currently soliciting feedback for an improved version of /review to help with the very important task of maintaining site quality. There are three targeted tasks we have built thus far and will continue to roll out others after we gain confidence these three are working:

  • Low Quality Posts: This task targets posts that we've algorithmically determined to be low quality. Your job is to read these posts and:

    • decide if a post does in fact Look Good
    • Edit the post to make it better
    • Delete1 the post entirely
    • or skip it if you are Not Sure.

    The minimum reputation to perform this task corresponds with the ability to edit any post (currently 2,000).

  • Suggested Edits: This task targets posts edited by users who have not yet earned the reputation to fully edit. Your job is to read these edits and:

    • decide to Approve the edit as is
    • Reject the edit
    • Edit the original post and make your own edit
    • Delete the post (moderator only)
    • or skip it if you are Not Sure.

    The minimum reputation to perform this task corresponds with the ability to edit any post (currently 2,000).

  • Close Votes: This task targets posts that have received at least one close vote. These posts generally need additional review to help decide if a post does in fact require closure. Your job is to read these posts and:

    • decide to Close this post by casting another close vote
    • Do Not Close2 this post if do not agree with the reasons for closing
    • Edit this post to make it better
    • or skip it if you are Not Sure.

    The minimum reputation to perform this task corresponds with the ability to cast close votes (currently 3,000).

Each of the actions on the review task listed above may not always be available based on your current reputation or moderator status. We will be reviewing the data collected in this experiment closely to determine if these new review tasks are working and also reserve the right to pull these tasks out entirely if we find they are not. We have enabled these review tasks on Meta, Super User, Server Fault, and Stack Overflow.

To check out the new hotness visit: http://meta.stackoverflow.com/review

1 If you don't have enough rep to delete, you can "Recommend Deletion". This doesn't do anything right now except move it to the front of the queue for people who can delete it.
2 "Do Not Close" just removes it from the queue right now -- it doesn't affect close votes.

share

locked by Shogging through the snow Aug 14 '12 at 15:25

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

4  
@nhahtdh: "These will soon be moved to the new beta Review Task system" - So yes. –  animuson Jul 12 '12 at 17:58
4  
What happens with "Not Sure?" What if you want to come back to it later? There doesn't seem to be a mechanism for un-dequeuing a post. –  CodeGnome Jul 12 '12 at 18:40
7  
What happened to badge progress? –  octopusgrabbus Jul 12 '12 at 20:25
4  
@octopusgrabbus you are still making progress. We will add this back to the new review task soon. –  Geoff Dalgas Jul 12 '12 at 20:27
9  
[cross-posted from SF.chat] You need 2000 rep to use the first two links. I have 1.4k rep on SF with 400 helpful flags. I'm going to still have to use the "old" links until they get move or I get more rep. I'm not expecting for the whole system to be changed but could they be made avalible to those with 2000k rep or 250+ helpful flags? That number demonstatres a decent understand of the flagging system. –  tombull89 Jul 13 '12 at 7:38
1  
Are there 85,3k CloseVotes or 85,3k questions with at least one close vote? The second metric would make more sense. –  örs Jul 13 '12 at 8:52
5  
Isn't a required rep of 2000 a bit excessive. Its not like a 1000 rep user wouldn't be able to tell if a question was low quality or not. –  VBwhatnow Jul 13 '12 at 10:52
6  
With such high rep limits, review might start stalling, especially on AskUbuntu, where panicky users are great in number, and returning users are few. –  hexafraction Jul 13 '12 at 12:28
5  
Just want to say that I really like the direction the review workflow has headed. Usability while reviewing was probably my biggest (of few) gripes with SO, so I'm super-excited. A huge thank you to the team for this. –  Rob Hruska Jul 13 '12 at 16:37
16  
The "not sure" option rubs me the wrong way. Makes me doubt myself sometimes when all I want to do is move on. Couldn't it say "Pass" or "Skip" instead? –  j08691 Jul 13 '12 at 17:27
1  
@PopularDemand - Yeah I had the same thought, although "not sure" connotes a meaning behind the action whereas "skip" or "pass" simply denote the action of moving onto the next review item. –  j08691 Jul 13 '12 at 18:44
4  
I like it, great idea!!! One comment: the close vote tab should have a big warning: CAUTION, USING THIS FEATURE CAN REALLY RUIN YOUR DAY (and an ad of the leading antidepressants brand). It's really depressing to see the amount of crap coming by when doing it for half an hour. Worse than the "low quality posts". Ehh, I guess it works... –  Gert Arnold Jul 15 '12 at 11:41
3  
@Gert: that's not a terrible idea... I know I can't spend too much time wading through the 10K/mod tools without sinking into despair. Maybe when you run out of close votes it should automatically redirect you to stackoverflow.com/questions/greatest-hits... er... –  Shogging through the snow Jul 16 '12 at 3:41
2  
I'm a <2K user and I like the old system as it is. I couldn't access to any of these new sections, only to the statistics. Why? That would be my "don't like, stay with old". And of course, I couldn't test any new feature. –  Shegit Brahm Jul 17 '12 at 7:01
1  
This way is far more effective and addictive than before. I find myself spending more time looking at the question, and spending more time than in the previous version! –  C. Ross Jul 18 '12 at 13:26

58 Answers 58

Is there any chance of going back to the review list upon saving an edit?

When a <2k user clicks on "More" under suggested edits why not show the pending edits without the option to interact? This would show what your <2k peers are editing and how (learn to edit by example).

share

If you don't have enough rep, the page just kind of...doesn't work. It's not immediately clear why I can't perform any of these tasks.

As a 500 Stack Overflow rep user, I see this:

Enter image description here

Ideally, there could at least be something explaining the requirements (and that I don't meet them) for each task I don't qualify for.

So apparently it does say you need at least 2k reputation to review suggested edits. Maybe make that text red (when I don't have enough rep) to draw attention. I literally did not see that when looking over the page.

share
4  
The message "You need at least 2k reputation to review Low Quality Posts" is only displayed if you're being locked out for that reason. However, this isn't clear; it just looks like it's a standard part of the description. It would help if the text were emphasized by being red or bold or something. –  Jeremy Banks Jul 13 '12 at 13:32
1  
@JeremyBanks hah, I didn't even notice that at all. Maybe make the text red. –  Ben Brocka Jul 13 '12 at 13:36
3  
The 2k requirement to ACT ON (detected) low quality posts is reasonable. But, it was one of the things that I used to browse periodically, both to flag non-answers as non-answers, but also to make edit suggestions of things I knew I was capable of describing more accurately or that I knew I could state more clearly what the person was trying to say. Is it possible that the "old" review system remain present for those users (like me) that can make use of just browsing the "low quality" questions/answers? –  killermist Jul 18 '12 at 20:38

The algorithm under the 'Low Quality Posts' retrieve too many items of the SQL section in my opinion.

SQL question are generally answered by a few number of characters representing the query, with little comments. All the answers are good, maybe they need more explanation, but that's it.

Maybe a filter by tag for this particular case should be an improvement.

Or mix the length of the post with the reputation of the poster. A 20 characters answer of a 2000+ rep user must be better than a 50 characters answer of a 10 rep user.

share
1  
Or, check the other answers. If there are 5 (n>1, really) answers on the same question, and all are "low quality" then it may be because that is the best answer warranted by the question. In this case, I'm all for skipping over these. However, as soon as someone has gone a little above and beyond the rest, then maybe the others should be more critically reviewed. –  Gaffi Jul 25 '12 at 17:53

In the old review-process I could restrict reviews to the tags I understand. In the new review I am supposed to give comments to things I do not understand.

Thus, I would like to have my tags back!


For clarification, these work in the old process:

These do not work in the new process:

share
5  
This was the first thing I noticed. I tend to limit my reviewing mostly to my areas of specific expertise, where I'm most likely to be able to locate duplicate questions, etc. –  Michael Berkowski Jul 13 '12 at 0:34
1  
This isn't so much a problem when reviewing for writing quality, but it's certainly a major issue when trying to determine whether a post is on-topic or not, and whether it should be closed or migrated. I want tags back too! –  CodeGnome Jul 14 '12 at 14:03
2  
+10 if I could. –  Bazzz Jul 19 '12 at 7:15

Users are locked out of reviewing Close Votes when they run out of close votes for the day.

Consequently, the Top Reviewers each day are the users with the greatest "do not close"-vote to "close"-vote ratio, because they are able to accumulate the most reviews before being locked out. These users also have an advantage in the all time Top Reviewers list -- I took an early lead, but lost it and continue to fall further behind because I cast fewer "do not close" votes than certain other users.

This current design will skew votes towards "do not close", or at least encourage users to select "not sure" instead of "close" to avoid getting locked out. Instead, "close", "do not close" and "edit" should all count towards a daily review limit.

(Why am I including "edit"? It's good to encourage edits, but once we include "do not close", having a larger number of "edit"s would be the only way to get an advantage and would be the new factor determining the daily Top Reviewers. This would lead to a lot of frivolous edits bumping questions that should actually be closed.)

After that, consider removing the daily Top Reviewers list. It will be a lot less interesting once everybody's tied at a shared cap.

share
9  
"Not Sure" really shouldn't be counting as a review - if you can drive yourself to the top of the leaderboard by abstaining then that's a bug. But yeah, "Don't Close" should be limited also. –  Shogging through the snow Jul 13 '12 at 15:28
1  
@Shog9 You're right, it doesn't count as a review. I just meant that users shouldn't be encouraged to press it when they actually believe that "Close" is appropriate. –  Jeremy Banks Jul 13 '12 at 15:32
1  
I'd be thrilled if this could be changed before my close:do not close ratio causes me to lose my (slightly unfairly-earned) position at the top of the leaderboard. –  Jeremy Banks Jul 14 '12 at 14:30

For the Suggested Edits tab, I don't have the 5K rep needed to access the Suggested Edits queue on my main site (SuperUser). I just see it as always 0 suggested edits needed. I do have more than 2K rep on that site, so the link does show up for me, but why is it there if I can't access the queue?

share

Here's a :

When voting to delete an answer with a 0-score, in that case, this one, it says that I voted to delete it in the history.

… but it doesn't show on the actual post:

The delete (1) link is missing here.

share
1  
Ah, that was actually one thing I had pointed out earlier in context of the "recommend deletion" confusion. Still it would be nice if the post actually showed delete (1) on the normal page just so it doesn't come across as a bug ;) –  slhck Jul 20 '12 at 22:27

While I'm not 100% positive it is a result of this new feature, as one who does not have the authority to edit posts directly, I have noticed a significant decrease in the time it takes for my suggestions to be reviewed. Generally within a few minutes, my suggestions are accepted, whereas a few weeks ago, it could have taken a few hours.

It also appears that the suggested edit queue has been kept very low, which also helps me, since I find it very frustrating to look at a horrible, needs-to-be-edited-now post, but I can't submit an edit because there are too many pending suggestions.

From my perspective, the new system works great!

share
3  
I think people are reviewing edits more now. I do believe that was a major goal for this, so it sounds like it's working fabulously –  Ben Brocka Jul 20 '12 at 14:06

One task I'd like to be able to do is look for old questions and answers that are under review. It was difficult under the old system, but impossible under the new. While not everyone is interested in this sort of drudgery, this clean-up can be quite helpful, given the number of old duplicates and poorly written questions that crop up when trying to search the site.

Some have requested a feature to limit reviews to certain tags. My suggestion would be for an expanded version of this, with various filters that can be applied to reviews: tags, dates, the "Close Votes" and "Close Reasons" from the old review system, flags (user- & auto-flags).

share

It would be great if an option can be provided to filter the answers/questions for review using specified tags.

I cannot see any more progress towards my gold badges, like Copy Editor, Marshal, Proofreader, etc. It will be handy to have one.

Also in the new review system a reviewer cannot upvote/downvote the answer being reviewed (and flagging).

It seems that classification of flagged questions is very confusing. I.e. the classification should be split into groups according to actions to be taken of the question: delete (not a real question, spam), close, move (not belongs here). 3 options should be sufficient.

share

Could you please rename "Improve" → "Edit" in the suggested edits? "Improve" and "Approve" are just too similar words.

Just now I approved an edit that I wanted to improve … and mark it as not helpful.

share

Proposed Edits on Accepted Answers

It should be much more obvious (color coding? different border? clown cars and sirens?) when a proposed edit modifies an accepted answer. The tiny "accepted yes" on the right-hand side doesn't really leap out; you have to make a conscious decision to check for it each time.

share
1  
+1, they could color the vote box like normal, which would help "see" this. –  user7116 Jul 17 '12 at 14:41

Prioritize flags

It would be nice to have if there was a way to filter out auto generated flags or some other way to prioritize flagged things. For example Low Quality Posts section has many posts auto-flagged incorrectly. This makes it a bit waste of time to go through all of them. The "contribute something useful" to "spent time" ratio is very low IMO on these.

I guess user flags are more likely to be accurate than auto flags so it would be nice if they are somehow prioritized and handled immediately.

Another idea is to also prioritize recently flagged posts. I guess in most cases it is much more valuable to immediately handle a recent flag rather then spend time on an 1 month old question.

Also auto-flager could be improved. One idea that comes to my mind is that when a question has many answers that are flagged because of length & content, those answers could be excluded from auto-flag?

An example of legit answer with many other legit(but flagged) answers:

enter image description here

share

If you take action on a suggested edit which has already been approved/ rejected, you get a popup saying (along the lines of):

This suggested edit has already been approved. Please visit the post and ...

However, after this, you are still sat on the same edit, albeit with the button you clicked removed.

enter image description here

What I would expect to happen is the controls being replaced by "Next" and that taking you to the next edit.

share

Comment on review for low quality posts.

This applies to low quality answers only.

As noted in other questions we need to be able to vote on these.

We need to be able to see other answers as the action differs depending on these. If there is another good answer then the shown answer should be deleted. If there are no other good answers then it has to be improved.

Re recommend deletion - under the old system the only way to get answers deleted was to flag for a moderator but flags get "declined - flags should not be used to indicate technical inaccuracies, or an altogether wrong answer" so this is a totally new way of doing things. (although I would also note that according to this Q&A the mods might not be correct or consistent here) Thus I think the recommend for deletion really needs a full explanation.

share
1  
We also definitely need a way to convert (or recommend to convert) to comment. Many bad answers could be at least half-decent and not entirely irrelevant comments. It also seems that at least some bad answers are replies to what is basically a bad question, so the root cause would lie with the question. Some tools to deal with the question could be useful on the page to handle low-quality answers. –  Monolo Jul 17 '12 at 10:58

The Suggested Edits page does not provide any way to get the context. For example I was reviewing a Suggested Edit that added a change to run a command in the original answer as root. To undertand whether that was warranted required reading the original question.

When reviewing Vote to Close questions, you can just click the question to get the context. This is absent in the Suggested Edits page.

I had to click the user, then activity, then find the question the edit was made to so that I could read the question and answer in context.

share

One thing I don't like about the new suggested edit list is that it's no longer a list.

It used to show a list of up to 30 suggested edits, which made it easy to quickly scroll through them looking for things I actually know something about.

The new list shows me just one suggested edit, and I have to explicitly skip it (by clicking "Not Sure") before I can see another. It makes reviewing a lot of suggested edits much more awkward.

An unrelated thought: Should there be an option to show suggested edits tagged with my favorite tags, or to push them to the top of the list? Those are the edits I'm much more likely to be able to judge. Posted separately here.

share
3  
+100. I really miss the list view. It made it much easier to locate the questions I am not qualified to review and skip the rest. Having to review the items one by one is much more time consuming. –  Leigh Jul 16 '12 at 22:20
1  
So far it still feels a bit tedious and awkward to me :/ Having to navigate through every item is much more time consuming than skimming a list IMO. –  Leigh Jul 24 '12 at 22:09

(I had posted this as an addendum to another answer, but I thought it should be separate.)

I'd like to have an option to show suggested edits tagged with my favorite tags, or to push them to the top of the list (or, if there is no list, to give them some preference). Those are the edits I'm much more likely to be able to judge.

share

In /Close Votes, I'd suggest to add the following functionality to the Do Not Close button: if the question gets closed, cast a reopen vote automatically.

share
4  
We're considering a couple possibilities for what "Do Not Close" will ultimately do. One option is for it to cast a reopen vote. Another is for it to cause the close votes to start aging and eventually be deleted. –  David Fullerton Jul 16 '12 at 21:57

Few comments:

  • Can the confirmation for delete please be removed? If not, provide an option. It's the /review route. I know a lot will be up for deletion. Having a confirmation is really annoying. At least skip the confirmation for Moderators
  • It would be great if all answers were visible, below the question. Having that helps to know whether it should be moved to a comment or deleted altogether
  • After approving/rejecting a suggested edit, the other button ( approve/reject) still remains. Fixed
  • Also, why aren't the voting arrows present on the posts? :/
share
14  
+1 on the voting arrows –  mgorven Jul 12 '12 at 23:30
4  
You must construct additional voting arrows –  Ben Brocka Jul 16 '12 at 1:48

Can we please have a feature to Improve wiki tag edits? (clicking more doesn't add the improve option).

For example, in the edit presented below, the spacing and punctuation aren't very good, but the content is sound, however I wouldn't approve it till it is improved, at the same time I don't want to reject it as its not really invalid:

missing improve option for wiki tag edits

Also, I think it would also help if under the sections for rejection there was a "Edit doesn't match tag" option, specific to wiki tag edits, as I've seen quite a few edits that don't match (either cause they really don't match or there is a misunderstanding of the tag), and typing a custom reason each time is a bit laborious, but at the same time I don't feel that it falls under "Invalid Edit".

share
1  
The Improve option is omitted here because it's a tag wiki excerpt, and you don't have permission to edit tag wikis yet. What would be a nice workflow however (possibly) would be to offer you the Improve option, and when clicked, route your improvement into the edit queue. –  Matt Jul 16 '12 at 10:41
5  
Editing tag wikis (without it needing to be reviewed) is a 20k privilege. I agree that it'd be nice for you to be able to suggest an improvement though; which is what the second part of my original comment was addressing. –  Matt Jul 16 '12 at 10:45

When you did your 50 daily "Close Votes", the link is "blacked out" and a hint is shown: "You have no more close votes today; come back in 3 hours.".

But if you did your 50 reviews of "Suggested Edits", there is no such message. Not before you click "approve" you will get the "Out of votes - you may only vote 50 times a day." notice. (Especially annoying as there is no notion when to will have some more votes again.)

share

I like this so far. However, while looking at the Low quality posts section, I had issues with it. The only options to choose from are Looks good, Edit, Recommend deletion, and Not sure.

What does Recommend Deletion do? Does it make a flag or something? I wouldn't mind having the flagging option somewhere on this page, I feel it's sometimes necessary. I may want to flag it as "Not an answer" sometimes, or point out advertisement.

share

The new process seems promising, but I'm concerned it implicitly moves the goal posts for deleting Low Quality Answers.

Under Jeff Atwood's year-old dictum, deletion is for very low quality answers, unlikely to be salvageable, embarrassingly bad stuff.

The new process encourages reviewers to click Recommend to Delete if the answer under review would require substantial editing to fix (retyping > 20% of the answer and significantly changing its character) while a competing answer has already gotten it right, one way or another.

This might be intentional and/or a good thing. I'm just saying it's a change in criteria, and possibly an unintended consequence. The issue may deserve an airing in its own thread, but I wanted to throw it out here as it's early days.

share
4  
This is why "Recommend Deletion" is fairly toothless at the moment - we really want to keep a sharp eye on it for a while before making it any more powerful. One would hope the majority of reviewers would abstain from recommending deletion of posts that aren't unsalvabeable garbage, but if folks are too delete-happy then we'll need to account for that. –  Shogging through the snow Jul 14 '12 at 22:06

Voting Up or Down is Missing

Under the old system, you could vote posts up or down while reviewing them. Since voting is one of the metrics tracked for the Reviewer badge, why is this ability not present in the new review interface?

share
1  
@CodeGnome: Please see meta.stackexchange.com/questions/140405/… –  Somnath Muluk Jul 24 '12 at 13:05

The interface lagged while I was trying to submit an edit from the Close Votes review page. The submit button appeared to be disabled after the first submission, but I think I pressed enter multiple times.

It looks like the client let me send the same edit multiple times, and the server accepted them all without collapsing it to a single edit:

revisions

share

May "Not Sure" be replaced with "Next"?

If I want to pass to the next post is not only because I am not sure, but also, e.g., because I am reviewing low quality posts, and I am shown a question that I would vote to close, but I cannot because I have already voted to close all the questions I could.

share

This new system has many advantages over the old one, I believe that the only thing I really (but really!) miss is the ability to sort by tag, and some sort of relevance.

A couple of observations, though:

In the low-quality posts section, when presented with a short answer, the problem may not so much be the answer, but the question. It would be useful to have a simple way to work on the question, for instance voting to close it.

I was presented with something like 75% (if not more) of really old answers to equally old questions, where most of them already had accepted answers. Do we really have to wade through tens of thousands of old answers that the community has already relegated to the bottom of the pages by not voting for them? I am all for cleaning up stuff, but this is a bit too much even for me.

A suggestion could be to order the answers presented for review so answers with no votes and for which there is another, accepted answer are presented last in the queue. Especially if there has been no activity on the question for several months.

Similarly, in the close votes section, I was presented with a long series of questions on subjects that I didn't really feel an expert in (the lack of tag filtering again) and which were quite old (nearly a year). Feels a bit strange to be asked to spend time and effort on so many old and forgotten (many didn't show any activity for the last 10 months or so) questions where I don't really feel I have the insight to, for instance, remove a close vote.

Being able to up- and down vote these old questions would be a nice compromise. I'd feel more at ease just voting for a question in a field where I don't feel expert enough to decide whether it should be closed or not.

Lastly, the wording of the close reasons is slightly confusing, if not downright off-putting. Often a question is just a bad question according to the site's standards and expectations (that questions should show some self effort, be researched, etc.). In several occasions "Not a real question" is not correct - they are real questions, just not good ones.

In the case of a question that has lingered for months, asked by a user with a rep in the single digits, even if one leaves a polite comment, chances are that nothing will happen. There should be a simple and accepted way to close these questions for what they are, not for some bogus reason (too localized, narq, etc.).

share

When relative times are used elsewhere, the full datetime is included as a title attribute. This isn't the case for the times in the recent review list, or the "asked"/"answered"/"actives" times beside posts. I'd like it to be included.

share

Current behavior:

Reproduction:

  1. Run out of delete votes.

  2. Try to delete yet another post.

  3. Get the above error, trying to delete from the question page itself and note I ran out of votes.

  4. Be forced to press Not Sure instead to get rid of it showing up again and again... >_<

Expected bahavior:

  • Low Quality Posts: Replacing "Delete" by "Recommend to Delete" when I ran out of close votes.

  • Close Queue: Replacing "Close" by either a flag or an appropriate message to stop the user.

share
2  
Same goes for "close" in low quality once you've ran out of close votes. –  slhck Jul 13 '12 at 15:02
2  
@slhck Questions shouldn't be shown anymore if you've run out of close votes; or the Close should become a flagging dialog (doesn't belong here) like for users below 3000 rep... –  Daniel Beck Jul 15 '12 at 21:19
1  
This has been fixed and deployed to production. –  Geoff Dalgas Jul 19 '12 at 17:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .