Periodically, the Community Team will evaluate the quality of a beta site. In addition to examining general site traffic stats and Meta activity, we check the content quality by selecting 10 random open questions with at least one answer from the past 30-40 days to see how their answers compare to other sites on the Internet. This all goes into our evaluation of when a site is ready to graduate.
As discussed in Podcast #30, we would like to explore options to get the communities directly involved in these evaluations. After all, you are the subject experts, and your evaluation would be more meaningful.
How can the community pull off a self-evaluation that would be meaningful? What would be the process? How would it be organized? Can we use meta and/or chat? The process and the results should be transparent so the community can learn from the experience.
Below is an excerpt of a sample evaluation from User Experience, and the type of information we are looking for.
- Symbol for "Swipe Left"
Better. Our answer is a variation on a standard - it's going to be either a point or a pointing hand. Searching for "swipe" or "drag" or "fling" is actually very poor, it's searching for "touchscreen motion" or similar that yields icons and the like that you'd expect. So we're actually very advantageous on that respect.
- Intuitive vs. Efficient: Which is more important?
Worse. We're a very condensed and consumable version of what information is out there. This is a hot topic in UX and so it's no surprise that there are presentations and even full books on the subject which you can find with a decent search. I'd go for Par if the answer had a lot more detail on the different approaches, as most results cover the area of our answer in just their opening paragraph.
- Usability - Testing Windows applications under different settings
Worse. This may be up to interpretation based on how you interpret the author's degree of intent for usability and accessibility. While the accepted answer provides a couple of nice points, it only provides a couple sufficient for a particular angle of accessibility. Another answer provides a link to more resources, which can also be found with cursory searches.
- Correct cursor for "content cannot be interacted with"
Par. We cover the good strategies succinctly yet directly with both the accepted answer and its follower. The advice itself is not too easy to find to begin with, but at the same time, neither is our question. The lack of ease of discovery is kinda hurting it.
- Why is there a header and logo at the top in a web application?
Better. Most places and searches will tell you that this is a common practice. More in-depth places will go into the details of why, but you don't find those places as easily when searching this specific aspect - it is usually a part of a greater treatise on element placement. So we get a nice boost by addressing the "why" while also being discoverable on the lone search.
- Is “<Object name>: <quantity>” notation acceptable in most popular localization?
Par. This is a hard thing to search for. It's hard to even find this question. This is yet again one of the things that is better covered in in-depth resources but not discoverable. Our answers do provide both examples and strategies. However, because it is also not very discoverable, it cannot be called "better" that much. Not even sure how that can be corrected.
- Is UX the same as Usability?
Par. This is a common topic of discussion that is easily found. We have a fair amount of varied analyses in our answers on the subject, which also go and reference other resources for private study. External resources provide the gamut of better and worse analyses than what we offer, variant on how you search for the subject. In the end, we're one of several good answers on the subject. Sometimes, being par is a good thing.
- How to design a form that mostly is read-only, but sometimes editable?
Better. Most search results on the subject will tell you how to save an editable form in a read-only format, or strategies of two separate forms. There are few resources that cover the union of these as desired in this question, and though the proposed strategies are identical to ours, the discoverability of ours is its boon.
- How can we communicate that UX work can't be done by just anyone?
Par. Arguments can be said about whether there is danger to hosting this kind of loaded question, but that is irrelevant to this analysis. We have a usable if brief analogy of the subject matter, but quite frankly you can get results left and right in the camps of "Anyone can do UX" and "Not everyone can do UX". Kudos to the question for being able to show up on either side, but due to the subject matter it cannot be honestly stated that the stance given is "better" or "worse".
- What research is there suggesting modal dialogs are disruptive?
...what, why is Stack Overflow showing up when I start my search query for this?
Worse. By nature, our existing answer is not actually providing an answer to the given question. It's just confirming what's already said in the question (which you can also find in rudimentary searches anyway) and saying that the user has no answer. A failure is terrible even if the competition isn't faring any better.
End results: 3 Better, 4 Par, 3 Worse.
When we have strong answers, we have strong answers. There is a lot of good work in various answers, both those covered in this analysis as well as brief surfs of the site. Sometimes it is hurt by low discoverability, though, but that isn't a direct fault of the site (but it's something users could look to improving when they get a chance). However, there are some places where there are just... weak answers that are not befitting the expertise and quality demonstrated in other answers. Incomplete, very brief or curt, and otherwise just being "sufficient" without being excellent.
We can come up with the sample questions. How can we turn those questions and your efforts into a useful and authoritative evaluation of the site? How would you approach the task of a quality check?