The question owner and answerers always get notified by comments, and this is to make sure that they have an opportunity to respond to those comments and use that information to improve the question or answer.
Also, consider that comments are also left by the community as guiding comments to teach and coach users on how to use the system. If you don't see these, then you don't benefit. It's not just moderators who moderate Stack Overflow. Most users learn how to use the site from comments left from other users.
Aside from these points, adding user preferences to Stack Exchange is a decision that, if were mine, I would avoid. It adds bloat to the software stack and makes it harder to maintain. Every time I've tried to add preferences to an app I was building, it's complexity tripled. When Jeff was still with Stack Exchange, he also made it clear that they don't typically do preferences, and I tend to agree with this based on my own experience.
With that said, I like the idea of knowing that you're notified when I leave a comment on your post. You should be notified, since it's your post. Of course, after you receive the notification, it's then up to you whether or not to ignore or respond to the comments. If someone keeps trying to get my attention, usually help vampires with more follow up questions on my answer, I respond once or twice and then just ignore the next follow up.
Also, note that in the global inbox, it tells you what post the comment is left on, so if you really want to avoid reading the comments, you can simply not click the link. Good luck!