After intensely studying the system's patterns and analyzing information from various posts, I've found that the system selects the close reason to be displayed in this manner. Once a question reaches five close votes, the top-level selection phase decides which one of the five top-level close reasons to display, and then if the outcome of that phase is Duplicate or Off-topic, the respective specific selection phase takes place to determine which specific duplicate targets or off-topic close reasons should be displayed.
Top-level selection phase
During this phase, only top-level close reasons are considered (Duplicate, Off-topic, Needs more focus, etc.). Any close reason-specific sub-reasons selected by close voters aren't considered in this phase (specifically, duplicate targets, or off-topic sub-reasons - canned, migration, custom). The following rules are in descending order of precedence:
- If a moderator closes the question, the top-level reason they selected will be the outcome of this phase.
- If a user with gold-badge instant duplicate closing powers closes the question as a duplicate, the Duplicate reason will be the outcome.
- If the Community user closes the question in response to the author of the question approving a proposed duplicate (can be a pending close vote or a recommend closure flag), the Duplicate reason will be the outcome.
- If a majority of voters pick the same top-level reason, that top-level reason will be the outcome.
- If there is no majority of voters picking the same top-level reason:
- if two voters picked the same reason while the others picked all different reasons, that same top-level reason will be the outcome.
- if two voters picked the same reason while two more also picked a different, same reason (e.g. two voted as "Opinion-based", two voted as "Off-topic", and one voted as "Duplicate"), then of the four voters who picked the same reason as another, the reason corresponding to the last one to vote of these four voters will be the outcome.
- if all voters picked different reasons, the close reason selected by the first voter will be the outcome. (This is the only possible case of these three sub-bullets on Stack Overflow; as it requires only three votes to close questions, cases where there are two votes for a given reason will be caught under the "majority" main bullet above.)
If the outcome of this process is not Duplicate or Off-topic (i.e. does not have any sub-reasons associated with it), that reason will be shown as the final close reason, and the overall process ends. If it is one of these, the respective specific selection phase takes place.
Duplicate specific selection phase
If the outcome of the top-level selection phase was Duplicate, the following process is used to determine which specific duplicate targets (question(s) that the question is a duplicate of) should be shown in the notice.
- If the author of the question clicked the Yes button in the author-only duplicate notice to close the question, any duplicate target(s) chosen by both duplicate close voters and users without enough reputation to vote to close casting recommend closure flags will be shown in the notice.
- If not, and the question was closed as a duplicate for any other reason (i.e. the top-level process produced an outcome of "Duplicate" for any reason other than the author clicking the button), any duplicate target(s) chosen by voters only (not flaggers) will be shown. This includes any target that a moderator or binding gold-badge voter picked.
Note that duplicate targets can be changed later on; this process simply documents the targets shown at the time of closure.
Off-topic specific selection phase
If the outcome of the top-level selection phase was Off-topic, the following process is used to determine which specific canned off-topic close reasons should be shown in the notice, whether or not the question should be migrated, or if no canned reason (the generic off-topic notice) should be shown.
For the purposes of this process, only the close votes where the close voter selected the Off-topic close reason count: votes for reasons other than Off-topic don't count.
There are three types of off-topic votes: canned reason votes, migration votes, and generic votes. Votes where a user voted for a custom comment reason (i.e. "I'm voting to close this question as off-topic because") are treated as generic votes for the above classification. These votes may change form later in the process, but they start off as above.
The process is as follows:
- If a moderator casts a migration off-topic vote, the question will be migrated to that site.
- If the migration is blocked or rejected, the below rules apply for the notice.
- If a moderator casts an off-topic vote for anything other than migration, the question will not be migrated, the bullet below will be skipped, and the below process will continue from the main bullet below it.
- If there are migration votes:
- If at least three of the migration votes correspond to the same site, the question is migrated to that site.
- Again, if the migration takes place, but is subsequently blocked or rejected, the below rules apply for the notice.
- On Stack Overflow, which requires only three votes to close a question, this effectively means that a unanimous consent of all close voters is required to migrate a question.
- If the process gets to this point without the question being migrated, or the question was migrated but was subsequently blocked or rejected, all migration votes are converted into generic off-topic votes, and this process continues. (This includes cases where there are the required number of migration votes, but not enough votes to migrate to the same site.)
- If there are at least two votes for the same canned off-topic close reason, or a single vote from a moderator, that canned reason will be selected and shown in the notice.
- This applies for all canned reasons. If there are two or more votes (or a single moderator's vote) for another canned reason, that one will also show in the notice.
- If there are not enough votes for any canned reason to be selected (excluding generic votes), the generic "Off-topic" reason will be shown ("update the question to be on-topic for [site]").
Note that this process does have a few holes/edge cases, where the notice you'd normally expect given the votes doesn't match the actual notice:
If a moderator closes the question as off-topic with a custom reason, and there are at least two votes for a canned reason, that canned reason will be shown in the notice instead of the generic one.
If a moderator chooses a canned reason while there are already two votes for a different canned reason, both of these canned reasons are shown in the notice.
If three users cast generic off-topic votes (including converted ones where they were originally migration votes but the question wasn't migrated or was sent back), and two users cast votes for the same canned reason, that canned reason (a minority reason) will be shown in the notice instead of a generic notice.
tl;dr: a moderator's off-topic vote, or a majority of generic off-topic votes from non-moderators, will not override other canned reason votes by other users. See a more detailed post regarding these behaviors.
(Note: some sites have canned off-topic reasons that are very similar to, or the same as, the generic off-topic close reason. Technically, votes for these are still considered canned reason votes, and not generic votes.)
Update (November 2019): The recent deployment of new post notices hasn't changed the algorithm for deciding which close notice to show from what it was previously, other than one minor display change: in the case of canned off-topic closures, the names of the specific users who opted for specific canned reasons are no longer shown; instead, only the reasons are shown.
If you have any clarifications, suggestions, or corrections you'd like to make, please comment, edit, or suggest edits.