It's more likely that the vote fraud detector spotted the bad upvotes and just removed them. It may have happened for more than one day, hence losing more rep.
It's annoying - particularly as it can interact badly with the rep cap in some situations - but it's best to shrug your shoulders and move on.
