I hear you: just today I had four upvotes to an answer I wrote over a year ago, and it would be awesome to have a way to find where the traffic driving those votes and others like them is coming from. I had just finished my own (now-deleted) feature-request and someone pointed me here (odd that your question didn't show in a search on "referrer").
Here are some additional thoughts:
To prevent referrer spam, this information should only be shown via an ajax request (those specifically asking for it) and only available to those with at least a certain amount of rep. If you want to get fancy, more rep could give you the right to see more history, though that is hardly necessary.
I also considered filtering the stored information: perhaps not showing/saving referrers from within the SE network or Google, but after some thought I think it's better to be able to also see Google searches leading to a question and to preserve the chain within the network if other questions are pointing to this one. Another rejected idea is that this could only be shown to those with a vested interest (ie: post) in a specific question, but that would end up encouraging wrong behaviors (junk posts just to get referrers).
To be useful, you would want to at minimum be able to see a list of referrers today so that you can perhaps find and interact with whatever source is driving this traffic while it's still relevant. Additionally, all links into a question page (including direct answer links) should be aggregated together and any significant referrer should be available (even if it's a little stale), where significant is defined as has driven at least some number of views to the question.