It's more likely that the [vote fraud detector][1] spotted the bad upvotes and just removed them. It may have happened for more than one day, hence losing more rep.

It's annoying - particularly as it can interact badly with the rep cap in some situations - but it's best to shrug your shoulders and move on.

  [1]: https://blog.stackoverflow.com/2008/12/vote-fraud-and-you/