-3

Everyone is familiar with the fake review problems particularly on the first-post review. I have a suggestion that might help with the problem particularly for the first post reviews. Impose a minimum amount of time spent on the page before being able to do something other than skip it.

(NOTE I am not talking about suggested edits.)

Couple of thoughts led to this:

I find it interesting that so many people are able to routinely complete 20 reviews per day by so early in the day where I might scan it many times and barely come up with more than two reviews to undertake. This seems to suggest we have more capacity then needed (fake or not).

A good review on a first post generally can't possibly be done in under a minute. Human reaction time to even click is about 215 ms... completing a review under that would be extremely suspect ( http://www.humanbenchmark.com/tests/reactiontime/stats.php ) Add in some time to actually check for duplicates and read the question.. and what do you know a minute has gone by.

While I'm certainly not a venerable veteran here, when I took cursory glances at a large number of first question reviews I found a large number of ok's on clearly bad things that should either have been rejected or edited first got +1'd, many times by users well above 2K.

Particularly with new users first posts I would imagine there is a larger tendency for asking a question which has already been answered. Not entirely their fault, they might not catch or understand the subtlety of what the real issue is or how to connect that to another poorly named question. This coincides with another problem in my view of a completely generic title that amounts to "why doesn't my code work".
If the reviewer spent more than a second on it, I would expect them to steer the issue towards a useful title, check for duplicates and flag it as a duplicate first.

Summarizing:

Many of these things are hindered a minimum amount of time was required to be spent on a review, and perhaps further if you put a minimum time between first-post reviews being completed (in some fashion dynamically tied to number of reviews in queue. (I won't even try to suggest doing this to other reviews yet)

Note: Updated title and tags per some feedback and missing emphasis points

Improve this question
2
  • 1
    "I find it interesting that so many people are able to routinely complete 20 reviews per day by so early in the day" Timezones, dear Adam. Timezones.
    – Dan Hanly
    Commented Sep 10, 2013 at 16:41
  • @danielhanly.com I'm extremely aware, I checked the timezone of the reviewers... Oh i get what yuo mean, they might not be logged in from where their country is... I.E. if I was working out of Singapore my country would still say US but i'd be 13 hours ahead.
    – UpAndAdam
    Commented Sep 10, 2013 at 16:54

2 Answers 2

Reset to default
5

While a very low average review time can be a strong indication of a bad reviewer, it isn't always the case. Of the roughly 700 people who completed at least three First Post reviews in the last week, only 145 of them averaged more than a minute per review. Generally, I use 7-8 seconds on average per review as my threshold for a potentially questionable reviewer, which is far below the minute you suggest.

The reason for the short reviews is generally that you can pick out spam or non-answers within a few seconds. You don't need a minute to process that. There are a lot of instances of this within the First Posts and Late Answers review queues.

All an artificial minimum review time would do is slow down rubber-stamp reviewers, not stop them. The audits and manual reviews by moderators have cut down on most of the really bad reviewers, but the developers always seem to be tinkering with new ways of identifying them. Manishearth has a suggestion for one such way to cut down on review abuse by allowing us to tag bad reviews, and my answer there supplements that with a way that we might be able to automate part of this process for those who approve spam.

To reduce the damage caused by bad reviewers in these queues, I believe a better solution is to require more reviewers per post. In my experience, this would prevent spam and other problematic content from making it through due to the improper review activity of one person. I would also allow us to identify these bad reviewers, if coupled with the other linked suggestion.

Improve this answer
2
  • Until reading your answer right now, I didn't know that there were problems with spam and other outright crap making it through the review queues. I gave up on everything but the suggested edit queue a long time ago, but after reading this post, I think I'll Skip through the other queues looking solely for really bad stuff to flag whenever I see items available to review.
    – Mark Amery
    Commented Sep 10, 2013 at 17:17
  • That makes infinitely more sense, and works more generically for all reviews, doesn't require blind stereotyping on times, and also gives a more direct path to correcting behavior. Thanks for the input Brad! I wasn't hard set on a 1 minute, obviously but even if you drop to 30 seconds or 10 seconds... you are always just moving a line which only slows automation down.
    – UpAndAdam
    Commented Sep 10, 2013 at 17:29
3

This already exists, and has for quite some time. It's only a few seconds, because there are plenty of legitimate reviewers that can review posts in only a handful of seconds. If it's increased much more than it starts to inhibit legitimate reviewers just as much as it does illegitimate reviewers. The goal in trying to prevent people from abusing the review system is to come up with features that make the life of those abusing the system hard, but without harming those using it properly.

Improve this answer
15
  • Then I'd suggest it doesn't exist in an effective manner since it doesn't seem to be deterring anything from my view, or I wouldn't have made the post. I appreciate the point of not hindering actual reviewers but you haven't even defined at what time amount it even hinders said reviewers... but again I also question how they can check for a duplicate issue within "seconds" Given that many people such as myself can't find 3 reviews to look at in the day, let alone complete 20, you didn't even consider excess capacity. In short your reply demonstrates you didn't actually read the whole thing
    – UpAndAdam
    Commented Sep 10, 2013 at 15:54
  • @UpAndAdam Just because you can't complete 3 reviews in a day doesn't mean it takes everyone that long. You're arguing that nobody should be reviewing items faster than one per minute. I'm saying that there are some, even a noticeable number of people, that will be able to do at least some reviews rather quickly. There are posts that are very obviously good, and it can be seen in just a few seconds; there are also posts so bad you can tell they need to go away in a few seconds. There are some in the middle that will take a minute or several, but not all.
    – Servy
    Commented Sep 10, 2013 at 15:59
  • @UpAndAdam As I said, problematic reviewers are indeed a concern, we simply need to use different means of inhibiting them besides an excessive rate-limiting, so that we can prevent abuse without inhibiting legitimate reviewers.
    – Servy
    Commented Sep 10, 2013 at 16:01
  • 3
    @Servy We have to remember that the OP is just over 500 on SO, and hasn't experienced the big queues like you and me have - I imagine it would make a difference in your thinking. We've been reviewing for orders of magnitude longer, and are maybe a bit faster at reviewing - but it's great if the OP is spending that much time on each review. I just can't afford to.
    – Undo
    Commented Sep 10, 2013 at 16:03
  • I never said that any of that nor did I even imply it; I never said how long I or others should spend on a review, how long the delay should be, and how many should be done. So I think you missed something. You presuppose harm without defining a rate at which it even becomes noticeable. And you do so without even addressing any of my key points. I don't spend more than 2-5 minutes on a review unless its major editing. I'm saying I don't even encounter things to review on first post list.
    – UpAndAdam
    Commented Sep 10, 2013 at 16:08
  • @Wooble The whole POST is about the first post queue. It's in the title and the description. I can't yet even look at the other queue cuz my rep isnt high enough.. but thanks for downvoting me anyways.
    – UpAndAdam
    Commented Sep 10, 2013 at 16:13
  • @UpAndAdam You suggested a rate limit of one minute, saying that it isn't possible to review a post in less time: "A good review generally can't possibly be done in under a minute" That's false, and I can say so from extensive experience both reviewing and looking at the reviews of others. I've seen plenty of legitimate reviewers doing a number of reviews at 2-5 seconds each (not for every single post, but for a large portion).
    – Servy
    Commented Sep 10, 2013 at 16:15
  • @UpAndAdam You've only reviewed a handful of posts, mostly over a short period of time. When the queue was first released it had a few tens of thousands of items in it. That most certainly is several orders of magnitude more items than it sees currently (since currently it stays in the 0-2 orders of magnitude, not 4-5).
    – Servy
    Commented Sep 10, 2013 at 16:19
  • 1
    @UpAndAdam You assume that every single question requires looking up a duplicate. It does not. There are going to be some posts that are going to take a fair bit of time an effort, and there are going to be some that are not. In fact, in a lot of the queues, there are a lot of items that take very little effort, either because they're so obviously correct, or so obviously bad. Part of this speed is indeed a product of practice, which you don't have. As for how long is too long, as I already said, this feature exists. You can see for yourself, but the time is about 2 seconds or so.
    – Servy
    Commented Sep 10, 2013 at 16:21
  • That's a good point and a much better MADE one; at the same time yesterday isn't today. I doubt anyone would have had time to notice or complain about too many fake reviews being accepted back then.. Needle in a haystack. Today I'd suggest using this to help stem the problem of duplicate posts and perhaps I should have made that louder in my suggestion.
    – UpAndAdam
    Commented Sep 10, 2013 at 16:22
  • @UpAndAdam Actually, people were complaining about poor reviewers much more loudly, more frequently, and justifiably (there have been a number of features added to inhibit poor reviews, several have been quite successful), than today.
    – Servy
    Commented Sep 10, 2013 at 16:24
  • Not sure how I can find out how it exists when I can't even find something to review lol. but I'll take your word for it; That said I don't dispute at all that non-first post reviews should be significantly shorter in most cases.
    – UpAndAdam
    Commented Sep 10, 2013 at 16:24
  • @UpAndAdam I said it right in the first sentence of my answer. It's a few seconds. I haven't gotten out a clock, it's somewhere in the 1-3 second range. Time it yourself (or look into the JS source code) if you're really that curious. I'm not talking about other queues at all. You said first posts, so I'm talking about first posts.
    – Servy
    Commented Sep 10, 2013 at 16:26
  • That's really interesting, no question to the justifiable part, I'd imagine that would lead to a wild-west free-for-all
    – UpAndAdam
    Commented Sep 10, 2013 at 16:27
  • let us continue this discussion in chat
    – UpAndAdam
    Commented Sep 10, 2013 at 16:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .