What is this about?

The close vote review workflow is currently like this:

  1. You are given a question.
  2. You read the question and decide whether to close it or not.
  3. If you click on Leave Open, goto 5
  4. If you click on Close, you are given a list of close reasons, along with how many people has voted on each close reason.
    • If you select the appropriate close reason and click on Vote To Close, goto 5
    • If you click on the cross or outside of the dialog box, goto 2
  5. You are done with this question.

I must emphasize that, the question is not closed when the reviewer click on Close, but only when the reviewer has chosen a close reason and clicked on Vote To Close.

The reviewer may sometimes need to see the wordings of the close reasons again to be sure, so the reviewer may click on the Close button before he/she has a decision. Even if the reviewer thinks that a particular question should be kept open, he/she can still click on the Close button to bring up the close reason list to see which close reasons other users has chosen.

Now consider how the current review audit work:

  1. You are given a question.
  2. You read the question and decide whether to close it or not.
  3. If you click on Leave Open, goto 5
  4. If you click on Close, goto 5
  5. If you clicked on what the audit expects, it shows "Congratulations", otherwise it shows "STOP! Look and Listen"

It is pretty obvious that clicking on the Close button yields different results for real review items and review audits. While you can click on Close to view the close reasons and existing votes for real review items without actually voting to close, you cannot do that for review audits because this will immediately cause you to fail the audit.

Also, sometimes the reviewer may actually search for duplicates, and when he/she try to vote to close as duplicate, he/she fails the audit, which is counter-productive.

My suggestions:

I think that the review audit items which expect you to vote to leave open should have consistent behaviour like real review items, so I request the close vote review audits to not be fail when the reviewer click on Close. Instead it should show the list of close reasons with one (or two) fake close vote(s).

In addition, care should be taken when assigning fake close votes with the following reasons:

  • duplicate of...
    This depends on how the system pick the duplicate question. If it is picked randomly, care should be taken to avoid similar questions to be chosen as the duplicate, unless the I don't understand button is implemented.
  • off-topic because... Other reasons
    Try not to create custom reasons which actually make sense to avoid confusions.
  • too broad
    There are probably questions of this type which is not yet closed, so the I don't understand button may also be needed.

Moreover, if the reviewer decided to close the question as a duplicate of an unlisted question, it should be considered as a real and normal close vote and recorded, then the question should be thrown into the close vote review queue. It should not be counted as a failed audit.

Due to the above changes, audit items which expect the reviewer to close them will also require the reviewers to choose the close vote (because if they don't, robot reviewer can always just click Close and then go back to click Leave Open to cheat through audits). I think that the audit should be passed no matter what reason the reviewer chose because the system cannot assess whether the reason chosen is absolutely wrong.

Close vote review audit flawed - fails without casting vote
"STOP! Look and Listen" audit tricked me

  • 2
    unnecessary complexity
    – user221081
    Commented Oct 29, 2013 at 14:51
  • 1
    @mehow making reviewers keep open a separate tabs with exact wordings, or memorize them, seems to be more of unneeded burden than adding few more lines of code to audit.
    – Mołot
    Commented Oct 29, 2013 at 15:13
  • @Mołot Anyone who's closed a few dozen questions should know all the close reasons ("exact wording" is not really the point - a reason that includes "asking for code" doesn't necessary apply only to questions asking for code, "a library, tool or favorite off-site resource" isn't broad enough to cover everything that that reason is for, etc.). Commented Oct 29, 2013 at 15:14
  • the point is you either want to Close or Leave Open. Once you decide to close that should be it - no way back + unsure always skip.
    – user221081
    Commented Oct 29, 2013 at 15:14
  • @Dukeling I'm not subject to audits, as they apparently don't happen on Drupal Answers, so I lack personal experience. If audits only happens after few dozen reviews, then you are perfectly right.
    – Mołot
    Commented Oct 29, 2013 at 15:19
  • 2
    Whenever I review close votes on SO, I usually decide carefully, so when I see close votes that doesn't make sense to me at the first glance I tend to see what reasons are chosen and how many votes are on each reason to help me decide, instead of just skipping that question or chosing to leave open. I believe I am not the only one who does this.
    – Alvin Wong
    Commented Oct 29, 2013 at 15:22
  • @mehow necessary complexity "the more I think about it, the more it looks like it is worth investing effort into coding tests that fail at the moment of unambiguously proven failure. It is a honorable goal to teach people subtle intricacies of moderation, but so far it looks like getting audits there does more harm than good..."
    – gnat
    Commented Oct 29, 2013 at 15:25
  • Already answered this question once. Having enough trouble producing reasonable audits - unless you have a concrete suggestion for implementing this, something that doesn't involve hand-waving away the difficulties inherent in turning a task already difficult to automate into potentially 7x difficult tasks, you're not suggesting anything new. Review audits - particularly for close/reopen - are sanity-checks, nothing more.
    – Shog9
    Commented Oct 29, 2013 at 15:51
  • 1
    And if I may make a suggestion: ignore existing close votes. There might not be any, and even if there are the reasons are listed at the top - if you don't agree with them before seeing how many other people have agreed, then you don't agree with them at all. Don't give in to peer-pressure.
    – Shog9
    Commented Oct 29, 2013 at 15:55


Browse other questions tagged .