After intensely studying the system's patterns and analyzing information from various posts, I've found that the system selects the close reason to be displayed in this manner. Once a question reaches the necessary amount of close votes, the top-level selection phase decides which one of the five top-level close reasons to display, and then if the outcome of that phase is duplicate or off-topic, the respective specific selection phase takes place to determine which specific duplicate targets or off-topic close reasons should be displayed.
(In April 2020, the close dialog uses the words "Community-specific reasons" instead of "off-topic", and the database name was changed to "Not suitable for this site". However, the system of choosing close reasons hasn't changed, and the close notices still use "off-topic", so I'll still refer to that reason as "off-topic" here so this answer is more readable.)
Top-level selection phase
During this phase, only top-level close reasons are considered (duplicate, off-topic, needs more focus, etc.). Any reason-specific sub-reasons selected by close voters (duplicate targets, or off-topic sub-reasons - canned, migration, custom) aren't considered in this phase. The following rules are in descending order of precedence:
- If a moderator closes the question, the top-level reason they selected will be the outcome of this phase.
- If a user with gold-badge instant duplicate closing powers closes the question as a duplicate, the duplicate reason will be the outcome.
- If the Community user closes the question in response to the author of the question approving a proposed duplicate (can be a pending close vote or a recommend closure flag), the duplicate reason will be the outcome.
- If a majority of voters picked the same top-level reason, that top-level reason will be the outcome.
- If there is no majority of voters picking the same top-level reason:
- if two voters picked the same reason while the others picked all different reasons, that same top-level reason chosen by two voters will be the outcome.
- if two voters picked the same reason while two more also picked a different, same reason (e.g., two voted as "opinion-based", two voted as "off-topic", and one voted as "duplicate"), then of the four voters who picked the same reason as another, the reason corresponding to the last one to vote of these four voters will be the outcome.
- if all voters picked different reasons, the close reason selected by the first voter will be the outcome. (This is the only possible case of these three sub-bullets on sites which only require three votes to close a question; on such sites, two votes for the same reason constitute a majority and so will be caught under that above main bullet.)
If the outcome of this process is not duplicate or off-topic (i.e., does not have any sub-reasons associated with it), that reason will be shown as the final close reason, and the overall process ends. If it is one of these, the respective specific selection phase takes place to determine which duplicate target(s) or which off-topic sub-reason(s) should be used.
Duplicate specific selection phase
If the outcome of the top-level selection phase was duplicate, the following process is used to determine which specific duplicate targets (question(s) that the question is a duplicate of) should be shown in the notice.
- If the author of the question approved proposed duplicates and the Community user bindingly closed the question on their behalf, the duplicate targets shown will be the specific one(s) the author indicated as being helpful to them. The author can choose targets chosen by both close voters and recommend closure flaggers (users without enough reputation to close casting flags to close).
- If not, any duplicate target(s) chosen by voters only (not flaggers) will be shown. This includes any target that a moderator or binding gold-badge voter picked. Only a single duplicate close vote for a given target is required for it to be selected.
Note that duplicate targets can be changed later on by gold-badge users and moderators; this process simply documents the targets shown at the time of closure.
Off-topic specific selection phase
If the outcome of the top-level selection phase was off-topic, the following process is used to determine which specific canned off-topic close reasons should be shown in the notice, whether or not the question should be migrated, or if no canned reason (the generic off-topic notice) should be shown.
For the purposes of this process, only the close votes where the close voter selected the off-topic ("community-specific reason") top-level close reason count: votes for top-level reasons other than off-topic don't count.
There are three types of off-topic votes: canned reason votes, migration votes, and generic votes. Votes where a user voted for a custom comment reason (i.e., "I'm voting to close this question because") are treated as generic votes for the above classification. These votes may change form later in the process, but they start off as above.
The process is as follows:
- If a moderator casts a migration off-topic vote, the question will be migrated to that site.
- If the migration is blocked or rejected, the below rules apply for the notice.
- If a moderator casts an off-topic vote for anything other than migration, the question will not be migrated, the bullet below will be skipped, and the process will continue from the main bullet below it.
- If there are migration votes:
- If a minimum threshold of the migration votes correspond to the same site, the question is migrated to that site.
- Again, if the migration takes place, but is subsequently blocked or rejected, the below rules apply for the notice.
- On most sites, the minimum threshold is a majority of all close votes (3 on sites requiring five votes to close questions, 2 on sites requiring three votes to close questions, or a single vote on sites which only require one vote to close a question). However, on Stack Overflow, a unanimous consensus of all three close voters is required.
- If the process gets to this point without the question being migrated, the question was migrated but was subsequently blocked or rejected, or the question ceases to be eligible for migration (turns 60 days old), all migration votes are converted into generic off-topic votes, and this process continues. (This includes cases where there are the required number of migration votes, but not enough votes to migrate to the same site.)
- If there are at least two votes for the same canned off-topic close reason, or a single vote from a moderator, that canned reason will be selected and shown in the notice.
- This applies for all canned reasons. If there are two or more votes (or a single moderator's vote) for another canned reason, that one will also show in the notice.
- If there are not enough votes for any canned reason to be selected (excluding generic votes), the generic "off-topic" reason will be shown ("update the question to be on-topic for [site]").
Note that this process does have a few holes/edge cases, where the notice you'd normally expect given the votes doesn't match the actual notice:
If a moderator closes the question as off-topic with a custom reason, and there are at least two votes for a canned reason, that canned reason will be shown in the notice instead of the generic one.
If a moderator chooses a canned reason while there are already two votes for a different canned reason, both of these canned reasons are shown in the notice.
If three users cast generic off-topic votes (including converted ones where they were originally migration votes but the question wasn't migrated or was sent back), and two users cast votes for the same canned reason, that canned reason (a minority reason) will be shown in the notice instead of a generic notice.
tl;dr: a moderator's off-topic vote, or a majority of generic off-topic votes from non-moderators, will not override other canned reason votes by other users. See a more detailed post regarding these behaviors.
(Note: some sites have canned off-topic reasons that are very similar to, or the same as, the generic off-topic close reason. Technically, votes for these are still considered canned reason votes, and not generic votes.)
Update 1 (November 2019): The recent deployment of new post notices hasn't changed the algorithm for deciding which close notice to show from what it was previously, other than one minor display change to the off-topic close reason: the names of the specific users who voted for a specific canned reason are no longer shown; instead, only the reason is shown.
Update 2 (April 2020): "Off-topic" has been renamed "Community-specific reasons" in the close dialog. However, the procedure for selecting the specific reasons hasn't changed, and the close notices still use the words "off-topic".
If you have any clarifications, suggestions, or corrections you'd like to make, please comment, edit, or suggest edits.