I don't think any such limit is needed. According to the answer here: How does comment voting and flagging work? there is already a limit:
You get 30 comment upvotes per day
If the comment upvoting was unlimited, it would be very reasonable to limit the number of votes via limiting the rate the comments can be upvoted. However, with this limit, I see only one logical explanation for 5-second threshold: To save the user from running out of limit too fast.
I'd personally prefer to decide myself how fast I'm going to run out of that limit. The only thing I'd like to have is to be able to check how many comments I have already upvoted, and/or to be informed that I'm running out of votes (such as when upvoting questions/answers).