If I heavily downvote a troll will the vote fraud algorithm rollback my votes or is this considered?
Perhaps it should take into account if other users also downvoted him to tartarus
What I would do is limit my anti-troll voting pattern to just a couple of really nasty posts, and let all the really old stuff stay as is. If the user is really all that bad the total effect of a bunch of us doing the same will get the point across. And you can still flag, too.
If a troll is particularly active, flagging for moderator attention is probably more effective than simple downvoting.