I do agree that the envelope needs some work, but I disagree that it should be removed. The basic problem with the envelope is the delay - it often takes ten or twenty minutes to register a response, and likely by the time it does light up you've already read it and responded. The other, related thing is that, even if you have responses, the envelope is not lit for the first visit to the site after a period of being away. Subsequent page loads show the lit envelope, but the first does not.
Also, not many people know that the envelope also lights up when your posts are edited, which can lead to people thinking that the envelope is lighting up inappropriately. This should be modified to be more obvious when you click the envelope.