43

This is a revisit of Are certain users favored the opportunity to review posts? that seems to have grown stale without any solid answer.

Just recently I was baffled by a user's ability to review a "First Post" almost 6 hours after it had been reviewed previously:

first post review queue

In my experience, certain review queues seem to be more inclined to "single-user reviews" (like First Posts and/or Late Answers), while others are more geared to "multi-user reviews" (like Close Votes and/or Suggested Edits). This does not seem to be the case here.

The "requirement" for such late, duplicate reviews are questionable and does seem privileged when considering the multiple examples in the First Post review queue (just even from reviews within the last month or so). Even being able to review already deleted posts (I highly doubt this was an instance of keeping the review window open for more than an hour and only reviewing it post mortem - first post reviews are dealt with far more quickly than most other review queues):

enter image description here

enter image description here

While these examples do not speak to the legitimacy of the reviews, it sure seems biased. Something is amiss...

Over at TeX.SE we have a far smaller community, so these situations are far more noticeable.

Edit: The following are all candidates for the above discussion:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

Improve this question
10
  • FWIW, I've reproduced this problem across several queues on several different sites, only the Suggested Edits queue being exempt (not giving details on what's causing it though).
    – animuson StaffMod
    Sep 29, 2013 at 16:09
  • @animuson: True. While the question mentions/focuses on the First Post review, I've also noticed it to have a broader scope.
    – Werner
    Sep 29, 2013 at 17:29
  • 2
    I can assure you, though, that the user isn't receiving any sort of "special treatment" that allows him to review more. I'd personally classify this as a bug, even an exploit. But again, I don't really want to give away details about it.
    – animuson StaffMod
    Sep 29, 2013 at 18:17
  • @animuson: In conversation with the user it doesn't seem like there was anything different in terms of their usage of the site (apart from using multiple computers). What I find strange though is the time difference between reviews. The review of a deleted post is particularly peculiar, so I really don't know what's going on.
    – Werner
    Sep 30, 2013 at 0:33
  • @animuson: I don't recall seeing this anymore. Do you know whether this is something that has been resolved? You mention that you "don't really want to give away details about it." So I guess you know what's going on and would therefore be able to or have addressed it.
    – Werner
    Apr 13, 2015 at 23:58
  • @Werner: Fresh example, nothing at all unusual going on.
    – Nathan Tuggy
    Dec 15, 2017 at 16:36
  • @NathanTuggy: That's an 8 minute difference. As you can see from my post, the user in question had been posting numerous late additions hours after the review had been done. It's highly unlikely that such chronic behaviour was as a result of "nothing at all unusual going on."
    – Werner
    Dec 15, 2017 at 16:58
  • @Werner: The fact that the system allows it at all without any special behavior means that, for someone who has unusual behavior for whatever reason that doesn't necessarily involve exploits, the system plausibly still allows it even after quite a while. In other words, this post appears to be half bug report, half accusation; but the bug report only reveals some of the anomalous behavior in the system, and the accusation of deliberate exploitation of any of the related bugs can reasonably be doubted.
    – Nathan Tuggy
    Dec 15, 2017 at 17:05
  • 1
    @animuson There really needs to be a stronger indication that exploiting this bug is not acceptable behavior and can get you suspended. Right now the only indication of this is your tiny comment (that doesn't even have any upvotes) that mentions "exploit". How about an answer at least stating that this is considered an exploit and may get you suspended?
    – Simon Forsberg
    Jul 12, 2018 at 8:35
  • 2
    This bug is possibly resolved in the next build.
    – Awesome Poodles
    Jan 8, 2019 at 1:32

3 Answers 3

Reset to default
8

We've had a "grace period" for reviews since very early in the life of the system: reviews could come in late and still be recorded, although generally without affecting the actual outcome of the task. This was done to allow for the fact that multiple people are often reviewing the same task simultaneously; while the system tries to prevent this, it does not disallow it, and in some cases having multiple simultaneous reviewers is advantageous: tasks that require multiple reviews to complete (close, reopen) can be completed more quickly if the same task is assigned to multiple reviewers at once.

However... We originally based this grace period on the time spent on the task by the reviewer: when the task was first loaded, we recorded the time, and then calculated the time spent upon submission of the review. If no start time was recorded, then no duration was calculated. This effectively allowed unscrupulous people to submit reviews hours, days, even months after the task was completed.

As of today, I've altered this logic to pin the grace period to the completion of the task itself: reviews submitted more than 15 minutes after the task has been completed will be silently rejected, regardless of how long the reviewer spent reviewing (or whether the system can even tell). This does not prevent abuse, but it certainly makes abuse more labor-intensive... And it also fixes a related bug: Not getting “credit” for Improve Edit

Improve this answer
4
  • So people can still submit a review (legitimately or otherwise) up to 15 minutes after the review has been deemed completed to accommodate for people spending time working on it even though it may have been completed? By rejected, do you mean that it just won't be recorded which, as far as I can tell, is used for the basis of tallying the reviews in any specific queue?
    – Werner
    Jan 8, 2019 at 20:32
  • Right, the result is simply discarded, @Werner. This only applies to the bit of information noting which review actions were taken; if a reviewer edits, votes, etc. those actions are still recorded normally, just as they would be if the reviewer was performing them on the normal question page.
    – Shog9
    Jan 8, 2019 at 20:35
  • Thank you, but I don't think the "silently rejected" part is helpful. Can we at least get a .notify or alert of "Too slow, sucka."?
    – Awesome Poodles
    Jan 8, 2019 at 21:16
  • We had something like that early on, @Awesome; folks just found it annoying. Consider that the common case here doesn't involve abuse, but rather opening a review and then getting a phone call or something and coming back to hit a button an hour later. A less annoying solution would probably be something like a "completed" page that shows the status of the review and how it completed - this could have some other utility as well.
    – Shog9
    Jun 21, 2019 at 16:33
11

It's possible to fudge the system... :-|

I found the following to be the only way to achieve this kind of behaviour. One essentially update the HTML to submit whichever code you want to associate with a review other than the default associated with "Next" when viewing the historic review.

It can be replicated under Chrome, but most likely in other browsers as well with some web-element editing capability/functionality/extension (like Firefox).

  1. Find an existing review that is already closed, say this one:

enter image description here

It's obvious that I'm unable to review this entry since I can only click "Next". It could be that I've already reviewed this entry, or that someone else reviewed it and the review is now closed.

  1. Right-click someone on the page and choose "Inspect element":

enter image description here

  1. In the inspection window, search for "review-actions" to find the identifiers submitted when clicking the "Next" button:

enter image description here

The HTML code represents the data-result-type submitted with clicking the "Next" button as being 254.

  1. Edit this specific element by right-clicking on it and selecting "Edit as HTML".

enter image description here

  1. Change 254 to 18, which corresponds to "No Action Needed".* Effectively this submits a "No Action Needed" response when clicking the "Next" button, rather than the default "Next" submission.

enter image description here

  1. Click "Next" and revisit the reviewed item:

enter image description here

enter image description here

The possible review action codes are: Skip = 1 (all); Approve = 2 (SE); Reject = 3 (SE); Delete = 4 (LQ); Edit = 5 (SE, LQ, CV, H&I); Close = 6 (LQ, CV); Looks OK = 7 (LQ, T); Leave Open = 8 (CV); Recommend Deletion = 9 (LQ); Recommend Close = 10 (LQ); I'm Done = 11 (FP, LA); Reopen = 12 (RV); Leave Closed = 13 (RV); Edit And Reopen = 14 (RV); Excellent = 15 (SSE); Satisfactory = 16 (SSE); Needs Improvement = 17 (SSE); No Action Needed = 18 (FP, LA); Reject And Edit = 19 (SE); Needs community edit= 20 (T); Flag = 21 (T); Override rejection and approve = 22 (SE, completed reviews only); Override approval and reject = 23 (SE, completed reviews only); Needs author edit = 24 (T); Next = 254 (all); I Understand = 255 (all, failed review audits).

The parentheses above indicate the review queues for which the action is valid. Other than this, no other validation is done (e.g. answers in LQ can be reviewed as "Close"): FP = First Posts, LA = Late Answers, SE = Suggested Edits, CV = Close Votes, RV = Reopen Votes, LQ = Low Quality Posts, T = Triage, H&I = Help & Improvement, SSE = Site Self-Evaluation (deprecated).

Improve this answer
10
  • I don't know how/if this could be changed in terms of the review submission interface to avoid this kind of thing from happening...
    – Werner
    Oct 28, 2015 at 20:20
  • 2
    Are you sure you need all this stuff? Why not keep your browser open for a while on a review item and then click the button. Won't that result that in the same?
    – Patrick Hofman
    Oct 28, 2015 at 20:26
  • @PatrickHofman: Yes.
    – Nathan Tuggy
    Oct 28, 2015 at 20:26
  • @PatrickHofman: The result would be the same if you happen to be able to have the review item open when it's still reviewable. Some of the instances I show in the question point to something more complicated than just waiting a long time.
    – Werner
    Oct 28, 2015 at 20:28
  • How did you discover this?
    – user343622
    Nov 8, 2017 at 17:05
  • @BillalBEGUERADJ: Discover what, exactly?
    – Werner
    Nov 8, 2017 at 17:08
  • The whole thing you described through all your answer (you got my upvote several days ago)
    – user343622
    Nov 8, 2017 at 17:11
  • 2
    @BillalBEGUERADJ: I frequented some review queue histories, because I had suspected some people didn't really review properly (clicking "No Action Needed" when there was clearly some needed. That's when I saw items being reviewed out-of-sync. I asked around, and even asked the person who managed those reviews. No-one (including the reviewer) could give a straight answer. So I looked at the HTML code and experimented with tricking the system. The reason why I think this was the eventual use-case for the late reviews: the late reviews were very late; no-one would keep windows open that long.
    – Werner
    Nov 8, 2017 at 17:20
  • Recently I even set almost all my rep as a bounty to understand this, no valid answer. I then re-asked again, and I was pointed to your marvelous answer that I practiced today just for testing. Anyway, thank you for sharing and congratulations for being that clever.
    – user343622
    Nov 8, 2017 at 17:23
  • 1
    @BillalBEGUERADJ: I probably happened upon my situation in a very similar one to yours. Some reviews turned up far too late for it to be a coincidence (keeping a review window open that long). Moreover, I attempted that - keeping the window open for a lengthy period - and while I was able to click on some review action, it wasn't recorded (if I recall correctly).
    – Werner
    Nov 8, 2017 at 17:29
4

I've seen this happen a number of times if I leave a review page open for too long. Most reviewers probably do that very seldom, but I wrote a couple of scripts that auto-load new reviews (since consolidated into Review Stalker Reloaded: A cross-site dashboard for reviews), and if I'm not actually at the computer, they could sit there for a long time before I come back to them. (These days the script takes steps to avoid this situation, so I'm triggering this much less often. However, I have been beaten to the reviewing punch by 8 minutes, possibly less.)

Anyone using this or another script, such as one they wrote themselves, could run into the same phenomenon without doing anything they would consider particularly unusual.

Improve this answer
9
  • 1
    This seems to be the more likely scenario for most users. I doubt anyone taking the time to mess with the HTML.
    – Patrick Hofman
    Oct 28, 2015 at 20:27
  • 2
    Still, this needs some server side fixing if possible.
    – Patrick Hofman
    Oct 28, 2015 at 20:28
  • 1
    I doubt this is what happened, as explained in the numerous examples in my post. Certain review lists on TeX - LaTeX are rare, and for a person to have made contributions of the nature outlined in the question is highly unlikely.
    – Werner
    Oct 28, 2015 at 20:31
  • 1
    @Werner: If there are other reviewers out there with hand-rolled auto-load scripts (they're not difficult to write) I would not be surprised if they occasionally auto-grabbed reviews by mistake before anyone else could and accidentally left them there long enough for this to show up. Can you elaborate on the specific examples that couldn't be caused by this?
    – Nathan Tuggy
    Oct 28, 2015 at 20:32
  • @NathanTuggy: This is surely a possibility, however minuscule. The reason for that I feel stems from the period within which this occurred and during the odd occasion when I was able to ask the individual what they did to achieve such "late reviews". Their response was that they were doing nothing special. My list of 100 could potentially be from a delinquent reviewer, and it might just as well have been from someone modifying the script submission.
    – Werner
    Oct 28, 2015 at 21:49
  • @NathanTuggy: I'd appreciate it if you could test your answer when you have an opportunity. Make sure the review counter steps after leaving it open on your computer, and post some screen shots that replicate the result (to show that your review was counted, and happened after the review was completed).
    – Werner
    Oct 28, 2015 at 22:11
  • @Werner: Here is a recent example from my history. I also tried going the other way (where another reviewer hit NAN and I edited), but that did not record anything special and it does not show up in my review history. I suspect editing, at any rate, will mess it up and prevent it from counting. And this is an example where reviewing (but not editing) counted.
    – Nathan Tuggy
    Oct 28, 2015 at 23:03
  • @NathanTuggy: ...and that first example review counted?
    – Werner
    Oct 28, 2015 at 23:07
  • @Werner: the first and third are in my history, so yes. (The second one isn't in my history, and there are exactly four from today in my history and the counter says 4.)
    – Nathan Tuggy
    Oct 28, 2015 at 23:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .