It's more likely that the vote fraud detectorvote fraud detector spotted the bad upvotes and just removed them. It may have happened for more than one day, hence losing more rep.
It's annoying - particularly as it can interact badly with the rep cap in some situations - but it's best to shrug your shoulders and move on.
