Yes, the system analyzes voting data and will effectively cancel votes it considers to be fraudulent.
See the "Vote Fraud and You" blog post for more details.
Note that getting a set of people to vote for each other is only one possible form of vote rigging. I suspect that more often than not, it's asymmetric with the target knowing nothing about it.
A few times I've reported "odd" voting where suddenly (in a quiet period) I've had 10 different answers (including old ones) receive a single upvote almost simultaneously. Usually after reporting this to the mods they've found that it's a single person who's decided to visit loads of my old posts and upvote them. This is effectively a form of bias, and the votes are cancelled.
The exact rules followed by the automated system have not been revealed, for obvious reasons. (Indeed, if SO is ever open sourced, it'll be interesting to see whether this means there's suddenly a lot more fraud which is done in a way designed to squeak by the system. I'd hope not, but you never know...