I'd notify the downvoter any time a downvoted question is updated or commented, without any option from downvoter to turn it off, except lifting their downvote or the answer being deleted.
Downvoting, in my opinion, should imply more responsibility than just losing a point of rep and forgetting.
Usually, if an answer is irrecoverably bad, it keeps being downvoted until the answerer gets the message and deletes his answer, which is not a problem.
But if there is heavy activity or heavy discussion on the answer, there certainly is something about it, which should force a downvoter to keep an eye on the question.
A couple of updates from the comments:
- Individual sites can opt in or out (this was posted when only Stack Overflow was around)
- It make sense to notify the downvoters only if the score goes up compared to what it was at the time of the downvote
- A debounce interval of a week or so can be added ("ok, it's still bad, but I'll ping you again in a week if the post keeps getting action")