With the introduction of new, simplified graduation requirements, we're preparing to shut down the venerable Community Eval review process. Years of waning participation, combined with a general lack of clarity as to how these mesh with the normal graduation process, have turned this system into a bit of an anachronism at best, and a waste of time at worst.
This leaves open the question: how should we be addressing the needs that motivated this system's creation in the first place... And do we even need to?
How can the community pull off a self-evaluation that would be meaningful? What would be the process? How would it be organized? Can we use meta and/or chat? The process and the results should be transparent so the community can learn from the experience.
At the time, we – the six members of the community management team – were manually reviewing every beta site periodically, checking traffic, user engagement, and Q&A quality. Our biggest concerns at the time, born from experience on some of the earliest sites, was that these young communities would devolve into content farms, discussion boards, or simply grow quiet and gradually decay. So every few weeks, one of us would grab an analytics report and a pile of random questions and try to read the tea leaves...
By the end of 2011, it was clear this was neither scalable nor even particularly useful; the results were secret, emailed to a handful of people within the company and quickly forgotten about. Meanwhile, Robert found himself explaining again and again that we were not going to be shutting down otherwise-healthy sites that were simply a bit small...
So one day, in one of our regular team meetings, we hatched this crazy idea: instead of quietly judging site quality in some secret star chamber and then trying to hand-wave away questions about our inscrutable process, why not ask the folks who might actually know something about the topic to take a good hard look at what they'd built and... Decide for themselves if they were really making the 'Net better.
A year and a couple dozen further experiments later, we built an automated process for this on top of the then-new review system...
Looking back at how this was automated, it seems clear that we'd already lost track of why we were doing it in the first place. What we wanted was an opportunity for the community to gather and engage in some collective introspection; what we got was a parallel voting system, the results of which were divorced from any discussion or long-term artifacts.
For internal use, it offered the seductive convenience of scanning a row of numbers. Whether they meant anything is anyone's guess, but they look serious. But as to whether they helped the community gauge its own progress, face up to persistent problems... It's instructive to compare the last “experimental” quality eval post on Japanese Language and the first automated site eval post a year later: slightly less voting, 100% less discussion. This became a pattern...
The problem with trying to kill two birds with one stone is that you often miss both birds and lose your stone.
- Our internal goal was to see how the sites stacked up against the greater Internet - was the site adding to it, or just imitating? Hence the instructions referring to Google searches and associated comparison.
- Our goal for going public was to draw the community's focus away from numbers they couldn't easily change and onto quality issues that they could. Hence the talk of closing and editing.
Nothing wrong with these goals, but nothing in the actual process guided reviewers toward either one - instead, it just generated more numbers. Intrepid members who did step up to discuss problems that they observed often just ignored both and took the opportunity to talk about whatever else was on their mind; in no instance did the system attempt to guide or correct anyone's chosen strategy – if you wanted to go down the list voting satisfactory for everything and then call it a day, you could... And many reviewers did just that.
In conclusion, we're not happy with how this system is working. It frequently fails to provoke any useful introspection, and perhaps worse, when it does prompt some thoughtful response these writeups tend to sit unaddressed and undiscussed next to a pile of numbers from folks punching buttons.
Do any of you still find these useful?
...and if so, how so? Is there a baby lurking somewhere in this murky water?
Is this sort of forced introspection even needed anymore?
A lot has changed in the past three years. There are more review queues, more opportunities to identify and tools to address problems, better exposure of meta discussions, and better up-front vetting for site creation itself. Custom off-topic reasons promote discussion and resolution on the questions themselves, and stats offer experienced members a bird's eye view of problems as they develop. And there's a new process for graduation and hopefully clearer criteria for site closures, hopefully reducing the fear and doubt that so often led to the site-stats fixation in the past...