Hats are fun because they are new and awesome. But after a few months of looking at the same image, it would get old and its appeal would wane. But for now they are still awesome! :)
I think the general sentiment behind this idea is a good one. More bling, am I right? However, we already have badges, reputation, and privileges, and I am not sure hats would be the next thing to add.
Your comment "modifications to your avatar" I think is a neat idea. It would be nice to use part of the already existing framework to hook some of that into daily operations.
Perhaps by month, there could be one (non permanent) possible gravatar modifier available. The achievement required to use it (only for that given month) could be listed in a SE blog post. The hooks in the framework to allow for a gravatar modifier are already available (as can be seen by the Winter Bash), so this would not take much time to implement.
The benefit would be that the SE team could have increased blog traffic, and be able to on a month by month basis have the community focus on the gravatar modification achievement. It would be fun to participate, and could help drive traffic to needed parts or features of the site (such as the review queue or a beta feature).