Now this is interesting to me...
I happened to get a random downvote shortly after you posted this, and decided I'd spend a minute looking into this loophole again. I identified a suspect and deleted the answer, then wandered over to that user's rep page, where I soon saw this:
which surprised me, because that seems to be lasting proof positive of a downvote.
This does, howeverHowever, appear to only be visibleI tried later to the subject ofreproduce this effect by deleting the downvote; I don't see a similarsame post, and saw no such entry on the rep page of the user who downvoted you, so it seems that this was just a hiccup. It would definitely have been a bug had it been reproducible.