Voting is anonymous. One can't see who voted where, the only public information is the total amount of votes, in the user profile. For downvotes it's not even showing the daily amount to prevent cases where one says "see, he cast only one downvote today so it must be on my post".
However, there is a way to see the time when user cast a downvote - his first only, but still - and this is via the critic badge page. It appears to be given pretty fast, every 5 minutes so for example now I know that vikram cast a downvote between 16 to 11 minutes ago, if I got downvoted I can compare the times and have a very reasonable "target" using this information.
Marking this discussion for now though as I'm not sure how to handle it. Do you think it's important enough to change something, or is it better to let it be?