I think however the answer is not complete, or barely a link, but at least it is the user's effort to contribute. And since the answer was removed, it's already invalid the effort. So it seems not necessary to decrease reputation to negative the contribution at all.
While this may feel like a great gesture, it's actually very harmful to the quality of the site. The expectations are very clear of what types of answers to submit. These sites don't just exist to make someone feel good about him or herself just because he or she posted a link, which just about anyone can do with some good Google skills. If a user can't take the time or put in the effort to produce a valid answer, then there is no reputation reward or even the reward of having that content persist for longer than it would take for community moderation to remove it.
Answers that are barely a link are not helpful to future visitors, especially if the link were to break, and since voting is focused on content and not people, the reputation score should be a direct reflection of the amount of quality contributions a user makes to the sites, not just lame, half-hearted efforts.
Remember, the goal of Stack Exchange is to provide expert answers to questions, and a link hardly qualifies as an expert answer, much less an answer at all. In short, to keep reputation, users should use editing as a tool to improve an answer to ensure that it helps future visitors and makes these sites a great resource of knowledge, not just a link farm.
As Aaron Bertrand mentioned, downvotes are retracted when an answer is removed. Likewise, incorrectly cast upvotes on a low-quality deleted answer are also removed. Again, reputation is a reflection of one's existing contributions to the network, with some exceptions for older material that meets certain criteria.