You should always remove the downvote if the answer improves; the amount of time it took shouldn't matter in my opinion. Downvotes are for bad questions/answers, upvotes are for good ones, no vote at all is fair if you don't feel the edit is upvote-worthy. After all, the point of voting is to help future readers judge the quality of the post; you're not just helping the poster, here. And sure, if someone leaves you a comment telling you that there's a problem with your answer, why not comment back and let them know you fixed it? I'd like to hear that; it makes me feel like leaving constructive criticism in comments is worthwhile when I see improvements being made.
The problem is that there's no easily-visible way to see when posts you've downvoted are corrected. Unless you're the one that left the comment and the OP replies to your comment, you aren't going to notice (well, unless you keep all downvoted posts open in tabs, and that's going to get overwhelming fast). Not all downvoters are going to comment, though. Some people just don't want to/have time to, and even beyond that, once one person has left a comment with the criticism you want to leave, all you have to do is upvote the comment to show your support. And a reply to that comment isn't going to notify those people.
So yes, downvotes should be reversed after good edits, and commenting back at concrit is fine (I'd even say encouraged). But this isn't really going to make a big difference unless you have some way of being notified when posts you downvoted are edited.
One way to address this can be found in the feature request that Nick Stauner commented on above, but it's been marked status-declined. It basically asks for some way to be notified when posts you dowvnote are changed. I'd like this to at least appear in a tab on your profile, if not as an outright notification, but it seems that won't be happening at this time (though that request is almost 5 years old; maybe it's time to revisit the issue?)