Why it bothers me? Sometimes I read a comment that is a good question, and, since it IS a good question, I want to read the next comment right away because now I'm dying of curiosity! Now, I'm satisfied, both a good question and a good answer, I want to upvote both!
A good solution that will prevent us being annoyed and abusers from ...abusing: Allow 3 comment upvotes in 15 seconds! (assuming the time from page load is >15sec, otherwise spammers/abusers get's 2 extra upvotes at no 'cost'.)
There, you have exactly 100% the same efficacy against spammers/abusers, and you do not annoy the poor curious readers with short attention spans for upvotes.