Some hats are asymmetric, as are some avatars. Rotation helps some, but really, we need the option of flipping the hat as well.
Just look at my avatar with the Sherlock Holmes hat. I want the looking glass over my eye!
(Actually, there's also the issues of hats assuming the avatar is a face looking forward while sometimes it's a face looking sideways).