Communities are fractal... And linked
I was happy to see you tackle this right away: the notion that Stack Exchange is a community overlooks an essential property of how communities form:

As you say, there are communities within communities, neither separate nor identical.
But there's another factor here as well: the interests which bring us together in the first place! There are C++ communities which participate on Stack Overflow, whose members are a part of that community and this one, but who also participate in the standardization community, in the education community, in their local interest groups... The Euler diagram of "Stack Exchange Communities" is itself contained in a circle which overlaps those of external communities, and those have as much - if not more - influence.

This is why working with C++ questions here can feel so very different from working with PHP questions, Python questions, R questions, Rebol or Red questions... It's not just the technology that's different - the people are different, the attitudes and expectations that they bring to the table are distinct in significant ways.
This is the challenge - and the opportunity! Because these connections work in both directions, Stack Overflow's culture has influenced many other communities beyond its own; ideas and approaches that've originated here can today be found in communities whose members have no experience using these sites at all... Just as Roman culture continues to influence societies that were never part of that empire, Stack Exchange's reach far exceeds its nominal boundaries.
One could argue this was always the goal - not just to provide a space for gardeners, but to change how gardening is done. But it doesn't have to be the intent or the goal; it is something that has and will continue to happen.
Community gardens
I like your analogy; community gardens are a wonderful, flexible concept, able to serve numerous purposes, bring in a wide variety of people, host a variety of different plants and scale from tiny marginal plots to vast fields.
This fits in well with the notion of overlapping communities of various sizes as seen here...
...But there's another reason to like the analogy: a garden needs constant care and planning. ESPECIALLY when the gardeners come in with very different abilities, skill levels, ideas as to what should be grown... You need a system for keeping order, ensuring that resources are allocated effectively, providing guidance for those who need it; if everyone tries to plant in the first plot while the rest of the garden lies fallow, nothing will be grown. This organization doesn't need to be complicated or forced - humans tend to form them instinctively when they don't exist - but it must exist, and the form that it takes will have a dramatic influence on what the garden produces.
And once such a government exists for the garden, it needs to have clear goals...
I love all of my children equally
This is my only big concern with your post:
Two equally important groups
That... Sounds good. But it isn't true. You may love all of your children, but... You probably don't treat them in exactly the same way; each needs different levels of care, attention, resources. The two groups you identify are both important, but... Their importance is not equal.
In fact... the groups are not distinct. They overlap, and people move between them with some regularity: there are plenty of folks who last contributed a decade ago but still read; there are plenty of folks who've been reading for years but only just became members of the community.
The critical difference between the two groups isn't the people in them... It is what they do. A garden which has just been seeded may not be of much interest to observers, but it will in time - provided there are people to tend it. But the reverse is not true: a garden with many visitors but no one to weed, water, and re-plant it will quickly lose its audience as well; the plants will die, the soil will blow away, the place will lose its value to both groups. Both groups are important: but only one is essential.
You recognize this in the last paragraph of the section:
If we define the community to be all users (visitor and contributor), I believe that we can very easily downplay the important role and valuable contributions of the gardeners in the past that we need them to continue into the future. Contributors and passive visitors are not interchangeable (though of course most contributors are also audience members).
Yes. I suspect we both remember what the landscape of the Internet looked like when Stack Overflow was first launched... Awash in "content farms" stuffed with material bought or "borrowed" from its creators, optimized for page-views. And completely dead. No one to update obsolete information, no one to correct mistakes, no one to answer new questions. Gardens filled with dried flowers. From the late '90s through the early '00s, we watched one formerly-thriving community after another succumb to this blight: bought out, drained of life, corpses left posed for the unwary reader to stumble over, a macabre wasteland.
Which brings me to the last reason I like the garden analogy...
A garden is never done
I've used farming or gardening analogies quite a bit over the years, from musings on the problems of scaling to cultural issues to sustainable growth. It's not just that I've spent a good chunk of my life working in that field... The task of growing things has some strong parallels to the task of nurturing groups of people.
Specifically: you aren't done. You're never done.
The workers, the plants, the land itself all need regular attention if you wish to still have a garden a day, a month, a year from now. Communities are much the same way: as long as they live, they will have new problems, new challenges, new opportunities that require care and tending.
The community garden strengthens its community by bringing the people together for work, providing shared goals and allowing its members to share in the fruits of their labor.
The plants themselves may wither in the fall, but the community rejoices in their harvest and begins work for the next season, digging new plots and welcoming new members; it is evergreen.