I have lost around 170 reputation points in serial up voting so far, which is more than what I had earned from up votes. If this is the case then what if any of the known persons do serial up voting purposefully to affect our reputation?
I checked your reputation history; you lost 170 reputation points.
You had serial voting up (60 reputation) in your profile dated June 8, 2012 which was detected by one of Stack Overflow's automated scripts. That's why there is a serial voteup reversal of 60 reputation on June 9, 2012.
After this reversal, you had again reputation reversal on June 26,2012 in two parts (-90 & -30 ). For this I checked your profile from the beginning and I found that you had many serial voteups in small small steps.
So the conclusion is all your false votes has been detected by the system and reversed. You have not done any hard work to gain those reputation points, except doing the false voting.
It is nice that you have reported here. If it was not done by you then don't worry. The system will detect that particular person itself. There is not permanent solution for this kind of problem still. See my question, Is there a more permanent solution for repeated serial voting by the same users?
But if it was done by you from another one of your Stack Overflow accounts then a moderator will take care of your account.
One suggestion (I got it from a moderator): just concentrate on questions and answers (as the site is for Q&A). The rest of the things (worries) will be taken care of by automated scripts and moderators.
No, the vote fraud detection script will only remove suspicious votes given, not penalize you beyond that point. The reduction in your reputation was entirely from the removal of fraudulent votes, not anything above that. This removal will also only affect votes given by the individuals who provide sizable streaks of votes for you, not votes left by any bystanders.
What might be confusing you is that the votes which were removed did not all come from the same day. You and three coworkers have been coordinating votes between yourselves for a couple of months now, and almost all of those votes were invalidated at once.
Please refrain from coordinating voting to artificially inflate the reputation of you and your friends, and tell your friends to stop this as well. Votes should be left based on the content of the post, not the person behind it.