I'd recommend adding notifications for when users comment back/write back in a chat. I tend to close the link or end up doing other things making me totally forget about the current Chat I'm having with someone about a question. It would be nice if we could be notified when user replies.
Otherwise, if we don't want to be notified we can just leave the chat.