There are precedents for closing down beta sites due to a lack of community involvement which had left the site looking like a ghost town, with experts drifting off due to the scarcity of good content. There's no precedent that I can remember for closing down sites for lack of quality except in the private beta stage, but this doesn't mean it'll never happen.
There is a precedent for significantly reshaping a site during the early beta: Programmers, which was defined to be for “tabs or spaces?” and “what's your favorite programmer cartoon?”, went through a period of being about subjective questions with interesting answers, and was then reshaped to be “for programmers when they're in front of a whiteboard”. The pressure came from part of the community (a minority, I think) with considerable support from On High.
The first place to raise concerns is on that site's meta site. Voice your concerns. Back them up — don't just say that the quality is bad, but explain why and how it could be better. Are the questions uninteresting? Are the answers bad? How does the site fare compared to other websites (not just Q&A but also wikis, forums, blogs, etc.) about the topic. Cite specific examples and statistics.
Then give it a little time. To take the concrete example of Puzzling, I think there's been a fairly sudden change in the dominant tone of the site. From the early beta to late October, it was predominantly a site about crafting and solving puzzles, where asking for solutions was only permitted for puzzles with an objectively-verifiable solution. In the past few weeks, the site has been very heavily dominated by riddle-type questions, where answers are subjective, of the next-number-in-a-series type.
I don't have statistics to show exactly this, but I do have statistics on a related phenomenon, which is answers that are entirely in a spoiler block (not counting many more answers where the sole non-spoiler text is something generic like “here's my solution” or “nice one”), which people tend to do when a question is a challenge to solve a puzzle and the answer is a solution. As of a few days ago, there were 2 such answers among the ~3000 first posts on the site, going up to 24 Oct. There were 175 such answers in the next ~2000 posts, in the period from 25 Oct to Nov 15.
So there's some evidence that the character of the site changed around late October 2014. Now let's look at meta traffic: 2 questions were asked in September, 8 in the period from 1 Oct to 20 Oct, 12 from 21 Oct to 31 Oct, 42 from 1 Nov to 17 Nov. It's fairly clear that the evolution on the main site did trigger quite a bit of meta discussion. The meta discussion is not over, there is no consensus that things are going in the right direction. It's far too early to conclude anything about where the site is going next.
Returning to the generic case, if you think that important points about Stack Exchange are not made or ignored in meta discussions, you may want to contact Stack Exchange and suggest they take a look. Stack Exchange staff does monitor meta sites but they don't have time to read every post so they might not notice everything.
If the meta discussion settles in a direction which you think still warrants your criticism, then it would be time to take the next step, which is to voice your concerns either on the main meta or to the Stack Exchange team or both. At that point, evidence should include not only data about the site (including both self-standing judgements and comparisons with other sites on the topic, preferably including quotes from expert on the topic), but also comparisons with other Stack Exchange sites and meta discussions.