There are many low-quality and good-but-bad-fit-for-the-site questions from newbies.
One important factor keeping you from immediately downvoting is that you want to be nice. A -1 is bound to put people off, especially if it's their first contact with the site. Maybe they didn't understand how the site works yet, and will improve the question once they get it.
I think that is a fine sentiment; more often than not, however, the question stays in its sorry state - either the OP didn't get it, thought it a good question, or lost interest.
How about introducing a conditional downvote that comes into effect 12 hours after casting
if the question was not edited
if the OP didn't post a comment to their question (because that's often where they add follow-up info)
That way, a user can immediately downvote a currently bad question, but still give the OP ample time to improve on it without the downvote being visible to them.
If the OP edits their question or posts a comment, the timed downvote vanishes.
UI-wise, this could be done in an obscure way, like pressing Ctrl while voting or something. This doesn't need to be an immediately discoverable feature, it's aiming at power users only. It could even be made a 3k+ or 5k+ feature.
This would probably add complexity in a major way (which is why I don't think it has much chance to be implemented) but to me, the increased motivation to downvote crap might justify it.