The total vote count (score) is denormalized, but the individual up/down vote counts are not.
So to display it on every post would incur 2 vote table queries * number of visible questions / answers. Our DB is fast, but the vote table is pretty massive, and not doing a query is always faster than doing it..
It's possibly something we could do on demand (as @hmemcpy notes), but as an "always displayed" it is a non-starter.
(Also: downvotes are still quite rare on Stack Overflow, so I'd say about 90% of the time, when you see a score, it is pure upvotes.)
EDIT: Jarrod implemented displaying total up/down votes today. This is currently deployed to meta and will be deployed to other sites in the evening PST later today. Simply click on the vote number itself to show up/down vote totals. Requires 1000 rep.