This is a revisit of Are certain users favored the opportunity to review posts? that seems to have grown stale without any solid answer.

Just recently I was baffled by a user's ability to review a "First Post" almost 6 hours after it had been reviewed previously:

first post review queue

In my experience, certain review queues seem to be more inclined to "single-user reviews" (like First Posts and/or Late Answers), while others are more geared to "multi-user reviews" (like Close Votes and/or Suggested Edits). This does not seem to be the case here.

The "requirement" for such late, duplicate reviews are questionable and does seem privileged when considering the multiple examples in the First Post review queue (just even from reviews within the last month or so). Even being able to review already deleted posts (I highly doubt this was an instance of keeping the review window open for more than an hour and only reviewing it post mortem - first post reviews are dealt with far more quickly than most other review queues):

enter image description here

enter image description here

While these examples do not speak to the legitimacy of the reviews, it sure seems biased. Something is amiss...

Over at TeX.SE we have a far smaller community, so these situations are far more noticeable.

Edit: The following are all candidates for the above discussion:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

  • FWIW, I've reproduced this problem across several queues on several different sites, only the Suggested Edits queue being exempt (not giving details on what's causing it though). – animuson Sep 29 '13 at 16:09
  • @animuson: True. While the question mentions/focuses on the First Post review, I've also noticed it to have a broader scope. – Werner Sep 29 '13 at 17:29
  • 2
    I can assure you, though, that the user isn't receiving any sort of "special treatment" that allows him to review more. I'd personally classify this as a bug, even an exploit. But again, I don't really want to give away details about it. – animuson Sep 29 '13 at 18:17
  • @animuson: In conversation with the user it doesn't seem like there was anything different in terms of their usage of the site (apart from using multiple computers). What I find strange though is the time difference between reviews. The review of a deleted post is particularly peculiar, so I really don't know what's going on. – Werner Sep 30 '13 at 0:33
  • @animuson: I don't recall seeing this anymore. Do you know whether this is something that has been resolved? You mention that you "don't really want to give away details about it." So I guess you know what's going on and would therefore be able to or have addressed it. – Werner Apr 13 '15 at 23:58
  • @NathanTuggy: That's an 8 minute difference. As you can see from my post, the user in question had been posting numerous late additions hours after the review had been done. It's highly unlikely that such chronic behaviour was as a result of "nothing at all unusual going on." – Werner Dec 15 '17 at 16:58
  • @Werner: The fact that the system allows it at all without any special behavior means that, for someone who has unusual behavior for whatever reason that doesn't necessarily involve exploits, the system plausibly still allows it even after quite a while. In other words, this post appears to be half bug report, half accusation; but the bug report only reveals some of the anomalous behavior in the system, and the accusation of deliberate exploitation of any of the related bugs can reasonably be doubted. – Nathan Tuggy Dec 15 '17 at 17:05
  • 1
    @animuson There really needs to be a stronger indication that exploiting this bug is not acceptable behavior and can get you suspended. Right now the only indication of this is your tiny comment (that doesn't even have any upvotes) that mentions "exploit". How about an answer at least stating that this is considered an exploit and may get you suspended? – Simon Forsberg Jul 12 '18 at 8:35
  • 2
    This bug is possibly resolved in the next build. – Awesome Poodles Jan 8 '19 at 1:32

We've had a "grace period" for reviews since very early in the life of the system: reviews could come in late and still be recorded, although generally without affecting the actual outcome of the task. This was done to allow for the fact that multiple people are often reviewing the same task simultaneously; while the system tries to prevent this, it does not disallow it, and in some cases having multiple simultaneous reviewers is advantageous: tasks that require multiple reviews to complete (close, reopen) can be completed more quickly if the same task is assigned to multiple reviewers at once.

However... We originally based this grace period on the time spent on the task by the reviewer: when the task was first loaded, we recorded the time, and then calculated the time spent upon submission of the review. If no start time was recorded, then no duration was calculated. This effectively allowed unscrupulous people to submit reviews hours, days, even months after the task was completed.

As of today, I've altered this logic to pin the grace period to the completion of the task itself: reviews submitted more than 15 minutes after the task has been completed will be silently rejected, regardless of how long the reviewer spent reviewing (or whether the system can even tell). This does not prevent abuse, but it certainly makes abuse more labor-intensive... And it also fixes a related bug: Not getting “credit” for Improve Edit

  • So people can still submit a review (legitimately or otherwise) up to 15 minutes after the review has been deemed completed to accommodate for people spending time working on it even though it may have been completed? By rejected, do you mean that it just won't be recorded which, as far as I can tell, is used for the basis of tallying the reviews in any specific queue? – Werner Jan 8 '19 at 20:32
  • Right, the result is simply discarded, @Werner. This only applies to the bit of information noting which review actions were taken; if a reviewer edits, votes, etc. those actions are still recorded normally, just as they would be if the reviewer was performing them on the normal question page. – Shog9 Jan 8 '19 at 20:35
  • Thank you, but I don't think the "silently rejected" part is helpful. Can we at least get a .notify or alert of "Too slow, sucka."? – Awesome Poodles Jan 8 '19 at 21:16
  • We had something like that early on, @Awesome; folks just found it annoying. Consider that the common case here doesn't involve abuse, but rather opening a review and then getting a phone call or something and coming back to hit a button an hour later. A less annoying solution would probably be something like a "completed" page that shows the status of the review and how it completed - this could have some other utility as well. – Shog9 Jun 21 '19 at 16:33

It's possible to fudge the system... :-|

I found the following to be the only way to achieve this kind of behaviour. One essentially update the HTML to submit whichever code you want to associate with a review other than the default associated with "Next" when viewing the historic review.

It can be replicated under Chrome, but most likely in other browsers as well with some web-element editing capability/functionality/extension (like Firefox).

  1. Find an existing review that is already closed, say this one:

enter image description here

It's obvious that I'm unable to review this entry since I can only click "Next". It could be that I've already reviewed this entry, or that someone else reviewed it and the review is now closed.

  1. Right-click someone on the page and choose "Inspect element":

enter image description here

  1. In the inspection window, search for "review-actions" to find the identifiers submitted when clicking the "Next" button:

enter image description here

The HTML code represents the data-result-type submitted with clicking the "Next" button as being 254.

  1. Edit this specific element by right-clicking on it and selecting "Edit as HTML".

enter image description here

  1. Change 254 to 18, which corresponds to "No Action Needed".* Effectively this submits a "No Action Needed" response when clicking the "Next" button, rather than the default "Next" submission.

enter image description here

  1. Click "Next" and revisit the reviewed item:

enter image description here

enter image description here

The possible review action codes are: Skip = 1 (all); Approve = 2 (SE); Reject = 3 (SE); Delete = 4 (LQ); Edit = 5 (SE, LQ, CV, H&I); Close = 6 (LQ, CV); Looks OK = 7 (LQ, T); Leave Open = 8 (CV); Recommend Deletion = 9 (LQ); Recommend Close = 10 (LQ); I'm Done = 11 (FP, LA); Reopen = 12 (RV); Leave Closed = 13 (RV); Edit And Reopen = 14 (RV); Excellent = 15 (SSE); Satisfactory = 16 (SSE); Needs Improvement = 17 (SSE); No Action Needed = 18 (FP, LA); Reject And Edit = 19 (SE); Needs community edit= 20 (T); Flag = 21 (T); Override rejection and approve = 22 (SE, completed reviews only); Override approval and reject = 23 (SE, completed reviews only); Needs author edit = 24 (T); Next = 254 (all); I Understand = 255 (all, failed review audits).

The parentheses above indicate the review queues for which the action is valid. Other than this, no other validation is done (e.g. answers in LQ can be reviewed as "Close"): FP = First Posts, LA = Late Answers, SE = Suggested Edits, CV = Close Votes, RV = Reopen Votes, LQ = Low Quality Posts, T = Triage, H&I = Help & Improvement, SSE = Site Self-Evaluation (deprecated).

  • I don't know how/if this could be changed in terms of the review submission interface to avoid this kind of thing from happening... – Werner Oct 28 '15 at 20:20
  • 2
    Are you sure you need all this stuff? Why not keep your browser open for a while on a review item and then click the button. Won't that result that in the same? – Patrick Hofman Oct 28 '15 at 20:26
  • @PatrickHofman: Yes. – Nathan Tuggy Oct 28 '15 at 20:26
  • @PatrickHofman: The result would be the same if you happen to be able to have the review item open when it's still reviewable. Some of the instances I show in the question point to something more complicated than just waiting a long time. – Werner Oct 28 '15 at 20:28
  • How did you discover this? – user343622 Nov 8 '17 at 17:05
  • @BillalBEGUERADJ: Discover what, exactly? – Werner Nov 8 '17 at 17:08
  • The whole thing you described through all your answer (you got my upvote several days ago) – user343622 Nov 8 '17 at 17:11
  • 2
    @BillalBEGUERADJ: I frequented some review queue histories, because I had suspected some people didn't really review properly (clicking "No Action Needed" when there was clearly some needed. That's when I saw items being reviewed out-of-sync. I asked around, and even asked the person who managed those reviews. No-one (including the reviewer) could give a straight answer. So I looked at the HTML code and experimented with tricking the system. The reason why I think this was the eventual use-case for the late reviews: the late reviews were very late; no-one would keep windows open that long. – Werner Nov 8 '17 at 17:20
  • Recently I even set almost all my rep as a bounty to understand this, no valid answer. I then re-asked again, and I was pointed to your marvelous answer that I practiced today just for testing. Anyway, thank you for sharing and congratulations for being that clever. – user343622 Nov 8 '17 at 17:23
  • 1
    @BillalBEGUERADJ: I probably happened upon my situation in a very similar one to yours. Some reviews turned up far too late for it to be a coincidence (keeping a review window open that long). Moreover, I attempted that - keeping the window open for a lengthy period - and while I was able to click on some review action, it wasn't recorded (if I recall correctly). – Werner Nov 8 '17 at 17:29

I've seen this happen a number of times if I leave a review page open for too long. Most reviewers probably do that very seldom, but I wrote a couple of scripts that auto-load new reviews (since consolidated into Review Stalker Reloaded: A cross-site dashboard for reviews), and if I'm not actually at the computer, they could sit there for a long time before I come back to them. (These days the script takes steps to avoid this situation, so I'm triggering this much less often. However, I have been beaten to the reviewing punch by 8 minutes, possibly less.)

Anyone using this or another script, such as one they wrote themselves, could run into the same phenomenon without doing anything they would consider particularly unusual.

  • 1
    This seems to be the more likely scenario for most users. I doubt anyone taking the time to mess with the HTML. – Patrick Hofman Oct 28 '15 at 20:27
  • 2
    Still, this needs some server side fixing if possible. – Patrick Hofman Oct 28 '15 at 20:28
  • 1
    I doubt this is what happened, as explained in the numerous examples in my post. Certain review lists on TeX - LaTeX are rare, and for a person to have made contributions of the nature outlined in the question is highly unlikely. – Werner Oct 28 '15 at 20:31
  • 1
    @Werner: If there are other reviewers out there with hand-rolled auto-load scripts (they're not difficult to write) I would not be surprised if they occasionally auto-grabbed reviews by mistake before anyone else could and accidentally left them there long enough for this to show up. Can you elaborate on the specific examples that couldn't be caused by this? – Nathan Tuggy Oct 28 '15 at 20:32
  • @NathanTuggy: This is surely a possibility, however minuscule. The reason for that I feel stems from the period within which this occurred and during the odd occasion when I was able to ask the individual what they did to achieve such "late reviews". Their response was that they were doing nothing special. My list of 100 could potentially be from a delinquent reviewer, and it might just as well have been from someone modifying the script submission. – Werner Oct 28 '15 at 21:49
  • @NathanTuggy: I'd appreciate it if you could test your answer when you have an opportunity. Make sure the review counter steps after leaving it open on your computer, and post some screen shots that replicate the result (to show that your review was counted, and happened after the review was completed). – Werner Oct 28 '15 at 22:11
  • @Werner: Here is a recent example from my history. I also tried going the other way (where another reviewer hit NAN and I edited), but that did not record anything special and it does not show up in my review history. I suspect editing, at any rate, will mess it up and prevent it from counting. And this is an example where reviewing (but not editing) counted. – Nathan Tuggy Oct 28 '15 at 23:03
  • @NathanTuggy: ...and that first example review counted? – Werner Oct 28 '15 at 23:07
  • @Werner: the first and third are in my history, so yes. (The second one isn't in my history, and there are exactly four from today in my history and the counter says 4.) – Nathan Tuggy Oct 28 '15 at 23:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .