I agree; but I think it should be included in the close votes queue as well for the same reasons.
When reviewing those queues without this ability I often come across comments that I would like to vote up. In the Reopen and Close queues this is, I think, especially important as people often comment as to why they are voting in the way that they are.
You can signal agreement with someone's reasoning for voting to reopen, without creating additional noise, by voting up their comment. This helps the reasoning for the vote to stand out and provides a different viewpoint to the original closers. The converse is partially true in the Close queue; a user may well have stated why they do not believe a question should be closed. If you agree with their reasoning then being able to vote up the comment makes it more noticeable to potential close voters.