And if this does happen, is it even a problem?
Problem?? Herding behavior is the very foundation of social media and the social web!
A while back I wrote about intermittent (random) reinforcement. But if voting appeared to be completely random, people would just post junk all the time. Community members still need to be told when and why they were given reinforcement, even if it's only given some of the time.
Herding behavior, information cascade, and all of the related phenomena are the counterbalance to truly random reinforcement. Advanced technical questions ask voters to make difficult decisions and those voters don't have the time or inclination to test and thoroughly evaluate everything. Most will simply eyeball it and maybe vote based on their immediate reaction; if other people have voted similarly, that makes them more likely to act on that instinct.
And because of that, there emerges a quasi-realistic picture of which questions and answers are actually "good", and what you need to change (or continue doing) to pick up more votes.
Without this, at the speed most questions fly by, votes on your answers would all be little onesy-twoseys. You'd get the same kinds of feedback on 1-line half-assed answers that you would on picture-perfect reference code.
Of course, this leads to several undesirable side-effects, most notably the bikeshed effect (soft/easy questions/answers getting the most votes). But on the whole, it is a good thing. It is evolutionary. It is how people learn to function within a group. During the early stages, people aren't ready to think and act independently; the only thing keeping them from fleeing is the knowledge or perception that if they do what everybody else does, they'll be safe from harm or ridicule.
Why do you think internet memes are so popular? It's the safest form of humour. It's guaranteed not to offend the majority of the community (although it will obviously irritate some). It's a way to feel accepted without actually being noticed.
Eventually most people grow out of this stage and do learn to think and act independently, but for many, that takes a long time. In the meantime, they do what they're biologically programmed to do and herd. (Some people never leave this stage - obviously, then it becomes a problem for them personally.)
The "wisdom of crowds" is far from perfect, but absent some sort of automated testing facility, it's about the best we presently can do, and that's largely thanks to herding. Herding doesn't just influence these sites; it defines them, and almost every other form of social media.
And for the most part - we shouldn't worry about it. It's part of how stable societies are formed.