While I tend to agree with the other answers here which say that you should not downvote, I want to indicate one exception: off topic (usually programming) questions here on Meta Stack Exchange.
Since MSE does not have its own dedicated moderators, and since the traffic here is pretty low, we (the active high rep users, I believe I can speak as one) tend to moderate the site ourselves and keep it clean as much as we can, without involving SE employees who have the diamonds.
Technically, 20k rep user can delete a closed question right away only when it has score of -3 or less. Hence, downvoting is required as part of the cleanup. That is the reason I downvote off topic questions here, even if they're "good" on their own, if asked in the proper place.
If I see a "good" off topic question here on MSE with score low enough for deletion I usually won't downvote further. But if I see a "bad" question, which I'd downvote anyway even if posted in correct site, I'd downvote here as well regardless of its current score.