Here's a storyboard depicting how an improved comment notification system could be implemented on SO sites...
///// STORYBOARD - Version 1 //////////////////////////////////////////////////////////////////////
Note - This is the original version. Please scroll down for version 2.
Frame 1 (v1-old) - User has entered comment and wishes to notify Jeff Atwood that a new comment has been added to the thread. This is referred to as "tagging".
alt text http://img694.imageshack.us/img694/4308/commentnotificationconc.jpg
Frame 2 (v1-old) - User has clicked "select user(s)" and brought up list of only commenters who indicated they are willing to be notified of new comments (by selecting "Notify me only if commenter tags me" or "Notify me of all new comments"). Notice that the first few words of the user's comment are included. This makes it easier to match up the right comment with the right person, and helps nullify any issues with two users having the same name.
alt text http://img686.imageshack.us/img686/4308/commentnotificationconc.jpg
Frame 3 (v1-old) - User has selected Jeff Atwood and clicked Apply. Now, the "Notify" line reads "Notify 1 user of new comment" and an orange status bar appears to alert the user to the fact that Jeff Atwood has been tagged.
alt text http://img248.imageshack.us/img248/4308/commentnotificationconc.jpg
Discussion
I believe this UI offers the best of all worlds. Users do not have to receive any notifications if they don't want to. Users can specifically tag users that are willing to be notified if the comment is directed at them. Just as with many forums, only one notification needs to be sent per thread until the user checks the thread, so it's not like anyone will be spammed with 20 notifications when a single thread receives 20 comments. They'll simply receive one.
The only disadvantages I see are some added clutter and complexity, and of course, someone has to program this.
Edit
Right after posting this, I of course noticed something I'd like to change right off the bat. The radio buttons should actually be available all the time, not just when posting a comment. This way, users could subscribe even to threads they didn't comment on, and they could change their notification preference at any time. If there's any interest, I can fix the storyboard to reflect this change, but for now, I hope this offers some good food for thought.
///// STORYBOARD - Version 2 //////////////////////////////////////////////////////////////////////
I think Kip is right that my original concept was too busy. Maybe this second attempt will be more "Stack Overflow"-like in simplicity and style.
Frame 1 (v2) - User has entered comment and wishes to "tag" Jeff Atwood so he will be notified of this comment. User clicks "tag users". Note: the checkbox ("do not allow users to tag me") could be changed to a friendlier "do not notify me of new comments", although this may not be as clear.
alt text http://img51.imageshack.us/img51/4308/commentnotificationconc.jpg
Frame 2 (v2) - User reviews list of commenters and clicks the checkbox next to Jeff Atwood.
alt text http://img41.imageshack.us/img41/8575/commentnotificationconcb.jpg
Frame 3 (v2) - Subscribe feature allows users who have not commented in the thread to receive notifications.
alt text http://img36.imageshack.us/img36/4308/commentnotificationconc.jpg
Frame 4 (v2) (see Frame 3 image)
The link buttons in Frame 3 will change depending on the situation:
Subscribed users who now wish to stop receiving notifications:
add comment | stop notifications
Users who have commented in the thread and elected to allow users to tag them (but haven't subscribed):
add comment | full notifications | stop notifications
Users who have commented in the thread but elected not to allow users to tag them:
add comment | full notifications | allow users to tag me
Discussion
This version is much cleaner and less cluttered than the original version, yet it still allows users a great deal of flexibility in what notifications they receive. By default, users will only receive notifications if they own the question/answer (just as it works today) or if they have commented on the thread and a subsequent commenter tags them. However, users can opt out of any notification, or, conversely, elect to be notified of all new comments. Even users who have not commented in a thread can subscribe (and receive notification). All of this functionality is available with only a single checkbox and link button added to the current SO comment editor, and one or two additional link buttons next to the "add comment" link button at the bottom of each comment thread. The "tag users" link button expands into a simple checkbox list that users can use to tag users (but onlyu those users are willing to be tagged).
This UI would minimize the annoyance of receiving unwanted notifications, it would allow users to tag other users even though aliases are not unique (or static) on SO, and, most important, it would improve the exchange of information between SO users.
///// ANOTHER IDEA - Threading //////////////////////////////////////////////////////////////////////
Threading provides some interesting advantages when it comes to comment notification, but there are definitely some issues with it as well.
How it would work
There would be one comment button for the overall thread, and one reply button for each comment. If you comment directly on a question or answer, your comment will appear "left justified" (lined up with the left margin of the page). If you reply to an existing comment, your comment will appear indented beneath the comment you responded to.
Notification
You receive notification if users comment on your question or answer (as usual) or if they reply to your comment.
Pros
- Comments are organized into a format that is easy to understand.
- Users are notified only if someone responds to something they wrote (either a question, answer, or comment).
- UI changes are limited to indenting and the addition of a single reply button per comment.
Cons
- Need a mechanism to handle the edge case of a comment hierarchy that is so deep that there is no longer sufficient space to indent.
- Also need a mechanism to handle deleted comments that have child comments, such as replacing the deleted comment with a placeholder that says "(comment deleted by author)".
- Complex hierarchies would become hard to visualize.
- Might encourage casual chatting, which would make it more difficult to find the critical information.
Edit:
As suggested by Peter Mortensen, I've added red annotations to each image in the Version 2 storyboard to make it easier to see what the descriptive text is referring to.