I just had this happen to me, too. Yesterday, someone voted me 100+ points in about a minute. Today, about half of that was axed. I think users who respond negatively to points being taken away somehow are thinking that the points being taken away is a bad reflection on the user him or herself, which leads to negative feelings and a negative response.
Really, though, the points are being taken away because of the dubious actions of someone else. At least some of the points should never have been there--another person put them there in a dubious way. Focus not on the fact that points were taken away from you, but that the dubious actions of someone else were at least partially corrected.
This should not be taken personally, unless you were involved in collaborating with someone to rig getting a bunch of points for yourself. Then you should take it personally: your hard work of scamming the system just got undone. Bummer.
But otherwise, you had nothing to do with this, someone else was screwing around and that was fixed.