The link to an answer doesn't have your user ID in it for tracking like the one from the question does, so it can't be tracked. Should it be possible? maybe - but the links would be even longer than they are now.
For example (for me - being user
135201 on meta), this question link is:
And the link to this answer:
Currently you see they have very different formats...so really it's also a discussion about changing the overall link format, which would be required to support awarding the badge for answer links.
Doing it yourself, it would look like this:
This is just a question link, but it scrolls down to the question with the hash just the same...so it's not tracked as an answer hit (if this matters), but will have the desired results: a tracked link to the question while still scrolling to the answer.