Only 20 reputation is required to be able to talk in the chat.
In the case were there are no downvotes, it only takes one of these combinations:
- Two answer upvotes
- Four question upvotes.
- One answer upvote and two question upvotes.
- One accepted answer and any kind of upvote.
- Winning a bounty.
If you vote the question of the user up, you limit this further; it only taken one of these combinations:
- One answer upvote and one question upvote.
- Three question upvotes.
- One accepted answer.
- Winning a bounty.
It's not really hard to get these votes, hence it is really not necessary to get temporary privileges.
To further outline this, comments should be replicated into both the answer and question when more details about the problem arise, and new problems should be put in another question. That way the comment thread can be cleaned by a simple flag and everyone can help to solve the problem. Personally helping the user outside the Q&A system hides useful information away from other readers...
Discussion in a chat is merely the result of a lack of detail in the question. Users need to update their questions to outline further details that were the result of answers, so that the question actually develops in the Q&A itself than in hidden away in the comments or on the chat. It benefits everyone...
Quality > Quantity.
Will there be an "invitation per SO question" feature for the chat? comes close as to another solution to your problem, where no privileges to the chat are given but only to a chat that's related to the question. It is status-planned and I see your question as a duplicate of that feature request...