This is something I have faced quite recently. Someone very new to Stack Overflow asked a question, but is not able to communicate properly or exactly understand the answers that has been given for his question. He normally clarifies that in comments. But sometimes, it is necessary to have a one-to-one conversation with the one who has given the answer, so that the comments section is not populated with some trivial comments. So, I just felt that it'd be great if the reputation limit for talking in chat is removed.
If you can't ask an intelligent & coherent question on the site, then I'm not really sure we want you participating in chat, either. Comments are great when you need to ask for additional clarification information, but when you get to the point of requiring an extended discussion in the comment section just to make sense of the question, red flags should start going up. This may not be a user who has found a good home on our site...
Yes, the dialogue in chat is a little less formal than the Q&A site, but the ability to participate in chat is a privilege that we only afford to certain users. Your proposal to "open up the floodgates", metaphorically speaking, is one that runs quite contrary to this notion and is a dangerous one in my mind.
And it's not that hard to gain chat privileges—only a nominal 20 reputation points are required. In other words, you need two upvotes on (an) answer(s). That seems like a sufficiently minimal bar to me. If you don't have any insights to share that are worth a measly two upvotes somewhere on the site, then we can probably afford to stand the chance that you'll take your online presence elsewhere.
Remember that all users start out at 1 reputation, including spammers. There's no way to tell a spambot from an actual contributor until they gain a few rep points. And spam is even more annoying in chat.