It would make voters much more accountable
No it wouldn't. Being able to harass someone isn't the same as being able to call them to account. If someone wants you to know why they voted, they can tell you that already; there's nothing you can do - even if you know the person involved - to force them to do so.
There are three pieces of information that are relevant to this discussion:
- Who voted
- How they voted
- Why they voted
You know #2 right now. #1 and #3 are not known to anyone but the voter himself. Of the two, only #3 is likely to be useful to you, and only if it pertains to something you could change on your post.
If someone has constructive suggestions for improvement, they can post those as comments. They can do this even if they can't or don't care to vote. If no one has done so, then there's no possible way for the system to expose that information anyway. So you're left with the choice of either looking for problems with your question yourself, or ignoring the votes and moving on.
You might also be interested in the information linked to from How to Ask.