Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 158 Stack Exchange communities.

What is meta?
Here's how it works:
  1. Any Stack Exchange user can ask a question
  2. The community provides support, votes on ideas, and reports bugs
  3. Your voice helps shape the way Stack Exchange operates

Ever since the suggested edit system was introduced, people have been consistently — if not loudly — griping that suggested edit reviewers need to do better jobs. After reading another question containing said gripe, it occurred to me that one possible way to identify problematic approvals would be to look at review speed. I emphasize "possible" because determining if this is true depends on data I don't have access to.

Judging suggested edits takes different amounts of time for different people, but there must be such a thing as a review that's too fast. Five seconds might be long enough to ensure that, say, two specific spelling fixes are okay, but there's no way it's long enough to look for other issues in the post (which is necessary to prevent "turd polishing"). That number might go up to 30 seconds or even a minute for a lengthy post containing numerous edits.

We've had a speed restriction on accepting answers for a while now. It might be good to have a similar one for judging suggested edits. The exact value would probably have to vary on a sliding scale based on the length of the post and the amount of editing that took place.

Could we get some numbers on how quickly suggested edit judgment occurs in practice? (What I really want is a graph of edit judgment speed vs. edit quality sorted by result (i.e. approval or rejection), but if we had a way to judge edit quality automatically, we wouldn't need edit reviewers in the first place.)

share|improve this question
As a very rough estimate, subsequent edits could be used to judge the quality of a review. – Bill the Lizard Nov 28 '11 at 18:35
I don't have access to an edit queue anywhere, but the one time I stumbled across an edit in need of reviewing, it was approved out from under me long before I had made my decision (to reject it). – John Nov 28 '11 at 18:40
Note if you click "reject" after it's approved, you are encouraged to go edit it yourself to make it better. Just did that today. – Kate Gregory Nov 28 '11 at 18:56
That is not a link to a gripe. – Matt Nov 28 '11 at 19:51
I was referring to the first sentence, which I'll copy here: "I've been doing a lot of edit reviewing lately and I seem to be the only one actually taking the time to analyze everything before accepting / rejecting an edit." @Matt – Pops Nov 28 '11 at 19:54
How would you even measure the time? I don't see how (short of a full eye-tracking setup) we could tell how long a reviewer has been reading a particular suggestion before acting. – balpha Nov 28 '11 at 19:54
@balpha, I was assuming that reviewButtonClickedTime - suggestedEditPageLoadedTime would be a decent approximation in a significant number of cases. However, I do like the idea of sending a full eye-tracking setup to every >2k user. – Pops Nov 28 '11 at 20:06
Sounds like an excellent use for the VC. Seriously though: The suggested edit queue page can have up to N+1 suggestions on it, so page load time isn't really a good indicator of when they started to read a particular post. – balpha Nov 28 '11 at 20:08
If this is deemed a useful set of measurements, perhaps a "study" UI could be rigged up which forced review of one suggested edit at a time. That would probably affect the results, though, since there would be no potential distraction from other waiting reviews. @balpha – Josh Caswell Nov 28 '11 at 20:36
Argh, I forgot that reviewers pass judgment on multiple posts directly from the queue page. I rarely see the page when more than one suggestion is in the queue. I also almost always go to the post's own page to see the full context before making a decision. But yeah, that does screw it up. @balpha – Pops Nov 28 '11 at 22:32

but there must be such a thing as a review that's too fast

If the review is good I don't see a problem

I would hate to see yet another nag-screen / popup when reviewing. (That's what you mean right?)

(What I really want is a graph of edit judgment speed vs. edit quality sorted by result (i.e. approval or rejection)

Would be interesting to see it.

share|improve this answer
Well, sure, fast approval of good edits is itself good. But how does a reviewer know whether an edit is good if he only spends two seconds looking at it? – Pops Nov 28 '11 at 19:45
@PopularDemand Familiarity with the question being edited, the post being really short, or the edit being simple typo corrections are three cases. You also have people like me who read abnormally fast. – Matthew Read Nov 30 '11 at 18:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .