The system allows votes to be undone within 5 minutes after casting. They can also be undone after an edit to the post.
This window is intentionally small to avoid people trying to game the system, but it exists at all because sometimes people make mistakes -- they read an answer, think it's good and upvote it, then read another that disagrees and realize they were wrong so they undo the vote. Or they're just fickle. Or clumsy. Or using a phone and that voting arrow isn't what they were aiming for. (I, like Richard Tingle in the comments, cast a lot of accidental votes on my phone and then have to undo them.)
If you do something that requires correction, you'll receive a comment (or an edit, or in an extreme case a message from a moderator). Votes are not punitive. And anyway, as noted in comments, removal of a vote isn't punishment in any case; it's not like you received a downvote.
Unless this happens a lot (two times is not a lot), it's best to assume somebody spazzed and just carry on. It's not personal.