I'm against it for a reason that I believe hasn't been mentioned yet, but I also will present a compromise.
While the difference of up- and downvotes is a useful indicator for the quality of a post, the number of upvotes alone is also. That is simply because often people are strongly disagreeing with each other.
On top of that, a downvote is often given not because a post isn't useful or even inappropriate, but merely because people didn't like it. When emotions run high, it can even deteriorate into a mere tug-of-war between two mobs. Given that the term disparaging is already in the question this scenario is already in the room.
During controversies, seeing both votes is most useful, only positive votes second, and the difference is the least useful of all options.
In fact in such situations seeing the standard deviation could be a better indicator than the difference, an indicator I often use to gauge the potential value of literature or movies on Amazon or imdb.
For posts on SO, those arguments apply rarely, but even on SO controversies do frequently occur in comment sections, so in those cases displaying a difference only may be harmful.
YouTube, for example, has such a voting system and suffers from the phenomenon of voting mobs trying to vote an enemy into obscurity (I presume that the proposal is to keep having a comment's score influence its prominence as it does now).
Disqus, on the other hand - a discussion system often used on blogs - shows both up- and downvotes, and in the face of controversies, it so manages to allow all sides of an argument to live while still displaying interesting posts more prominently.
I understand that SO isn't a political blog, but there are still arguments, and people are as irrational when feeling attacked here as anywhere else. In that light I feel that such a proposal would increase bad blood.
A compromise could be to display the difference but use only the upvote count to determine how prominent the comment is placed (ie. wether it will be visible before one clicks show all comments). That would take a significant amount of incentive away for voting people down to silence them while still showing the average visitor sentiment.