I disagree. For 2.5 reasons...
First and foremost, it'd be confusing. Right now, the number to the right means votes. It always means votes. You don't need to remember what your sort order is set to, you just glance to the right and there it is - the vote-count.
You could mitigate that by adopting a question-style rendering, with votes and view-count always displayed to the right. Maybe throw in the creation and last-modified dates to avoid the inevitable "recent and newest sort orders should tell you the dates" feature-requests.
But that'd be a waste of space. Look at how much room the question list takes up - but only a few questions are displayed per-page.
Lastly, the number is guaranteed to be inaccurate for answers, potentially very inaccurate. As Jeff notes, we don't actually track the views for answers - you assume when someone loads the page, that's a view for the question, but answers might not be seen, might be posted much later, might end up on a different page... It'd be extra effort to track, and probably still wouldn't be all that accurate. Good enough to sort by, but nothing more. And for what? So folks can fuss over another number? There are better uses of your time.
But yeah, I mostly just don't want a bulky list, or a number I have to think about to know what it means.