May I suggest a compromise?

I think that the information which is not available publicly should be served over secure connections. These can include:

 - when a moderator visits a page which is only accessible to moderators 
 - when a user visits his/her own userpage
 - when a user logs in using the log-in page

This shouldn't create a performance problem since the number of visits to these pages is limited. Also this is consistent with what large companies like Google used to do: user account setting pages were served over secure connection even when the https was not default for GMail.

Let me add that I think these pages *should* be served securely since they contain private information about users and SE should make sure that the information on these pages remains private.