I just accepted an answer to one of my questions on SU and wanted to throw the guy an upvote as well, but I was denied.
As you can see, I haven't cast a vote yet, but I am not allowed to do so because it thinks I have already.
You probably voted on it and removed your vote some time in the past. The timer starts ticking as soon as you vote, even if you remove that vote before it expires.
Touching the up or down arrow means you have a very short time window to decide what your permanent vote will be: up, down, or nothing.