Strategies for handling beta community quality evaluations

Question

Periodically, the Community Team will evaluate the quality of a beta site. In addition to examining general site traffic stats and Meta activity, we check the content quality by selecting 10 random open questions with at least one answer from the past 30-40 days to see how their answers compare to other sites on the Internet. This all goes into our evaluation of when a site is ready to graduate.

As discussed in Podcast #30, we would like to explore options to get the communities directly involved in these evaluations. After all, you are the subject experts, and your evaluation would be more meaningful.

How can the community pull off a self-evaluation that would be meaningful? What would be the process? How would it be organized? Can we use meta and/or chat? The process and the results should be transparent so the community can learn from the experience.

Below is an excerpt of a sample evaluation from User Experience, and the type of information we are looking for.

• Symbol for "Swipe Left"
  Better. Our answer is a variation on a standard - it's going to be either a point or a pointing hand. Searching for "swipe" or "drag" or "fling" is actually very poor, it's searching for "touchscreen motion" or similar that yields icons and the like that you'd expect. So we're actually very advantageous on that respect.

• Intuitive vs. Efficient: Which is more important?
  Worse. We're a very condensed and consumable version of what information is out there. This is a hot topic in UX and so it's no surprise that there are presentations and even full books on the subject which you can find with a decent search. I'd go for Par if the answer had a lot more detail on the different approaches, as most results cover the area of our answer in just their opening paragraph.

• Usability - Testing Windows applications under different settings
  Worse. This may be up to interpretation based on how you interpret the author's degree of intent for usability and accessibility. While the accepted answer provides a couple of nice points, it only provides a couple sufficient for a particular angle of accessibility. Another answer provides a link to more resources, which can also be found with cursory searches.

• Correct cursor for "content cannot be interacted with"
  Par. We cover the good strategies succinctly yet directly with both the accepted answer and its follower. The advice itself is not too easy to find to begin with, but at the same time, neither is our question. The lack of ease of discovery is kinda hurting it.

• Why is there a header and logo at the top in a web application?
  Better. Most places and searches will tell you that this is a common practice. More in-depth places will go into the details of why, but you don't find those places as easily when searching this specific aspect - it is usually a part of a greater treatise on element placement. So we get a nice boost by addressing the "why" while also being discoverable on the lone search.

• Is “<Object name>: <quantity>” notation acceptable in most popular localization?
  Par. This is a hard thing to search for. It's hard to even find this question. This is yet again one of the things that is better covered in in-depth resources but not discoverable. Our answers do provide both examples and strategies. However, because it is also not very discoverable, it cannot be called "better" that much. Not even sure how that can be corrected.

• Is UX the same as Usability?
  Par. This is a common topic of discussion that is easily found. We have a fair amount of varied analyses in our answers on the subject, which also go and reference other resources for private study. External resources provide the gamut of better and worse analyses than what we offer, variant on how you search for the subject. In the end, we're one of several good answers on the subject. Sometimes, being par is a good thing.

• How to design a form that mostly is read-only, but sometimes editable?
  Better. Most search results on the subject will tell you how to save an editable form in a read-only format, or strategies of two separate forms. There are few resources that cover the union of these as desired in this question, and though the proposed strategies are identical to ours, the discoverability of ours is its boon.

• How can we communicate that UX work can't be done by just anyone?
  Par. Arguments can be said about whether there is danger to hosting this kind of loaded question, but that is irrelevant to this analysis. We have a usable if brief analogy of the subject matter, but quite frankly you can get results left and right in the camps of "Anyone can do UX" and "Not everyone can do UX". Kudos to the question for being able to show up on either side, but due to the subject matter it cannot be honestly stated that the stance given is "better" or "worse".

• What research is there suggesting modal dialogs are disruptive?
  ...what, why is Stack Overflow showing up when I start my search query for this?
  Worse. By nature, our existing answer is not actually providing an answer to the given question. It's just confirming what's already said in the question (which you can also find in rudimentary searches anyway) and saying that the user has no answer. A failure is terrible even if the competition isn't faring any better.

End results: 3 Better, 4 Par, 3 Worse.

When we have strong answers, we have strong answers. There is a lot of good work in various answers, both those covered in this analysis as well as brief surfs of the site. Sometimes it is hurt by low discoverability, though, but that isn't a direct fault of the site (but it's something users could look to improving when they get a chance). However, there are some places where there are just... weak answers that are not befitting the expertise and quality demonstrated in other answers. Incomplete, very brief or curt, and otherwise just being "sufficient" without being excellent.

We can come up with the sample questions. How can we turn those questions and your efforts into a useful and authoritative evaluation of the site? How would you approach the task of a quality check?

Answer 1

We should try to keep the reviews independent, avoiding that the different reviewers from the community can see what an earlier reviewer wrote. I think we would get higher quality data that way. To achieve that the questions should be sent to the reviewers via mail, not posted publicly at first. After everyone has finished the review or some time limit is reached the reviews should be posted to the site meta.

I'd keep the identity of the specific reviewer secret, unless a reviewer chooses to void that anonymity. That might prevent silly stuff like revenge downvoting the reviewer if they reviewed one of your questions unfavorably.

I'm a bit skeptical if members of the community will actually review the site objectively. The users most likely to volunteer for reviewing are probably heavily invested in the site, they might not necessarily be objective. They will have more domain-knowledge than the SE employees performing the reviews now, so that might also produce some additional insight though. But I'm not sure if community reviews can completely replace reviews by people not that invested into the specific site.

Answer 2

Here's a basic idea I've been working on…

Submit evaluation request as a meta post with a list of the questions and some general instructions. Mark it community wiki for the widest possible participation.

Users would edit the original post inline with their evaluations (and better/worse/same ratings)… all within the question itself.

The answers would be relatively free form. They could be issues raised about any of the sample questions (or issues with the evaluations)… or even a broader evaluation of the site itself. Folks can vote on the validity of the statements and use the comments for a light-weight discussion. The end goal is migrate relevant answer-discussions into the original post/evaluation.

Hopefully the original post will morph into the finished evaluation. It's easy enough for us to review the "answers" and revision history to consider the commentary surrounding the final evaluation.

Possible caveats:

• Possible Biases Within the Community — The end result should be somewhat authoritative and the tendency will be for the community to want their reviews to be universally favorable. Will users step up as the sacrificial lamb when a poor review is warranted? The term "kill the messenger" comes to mind. This should not become a divisive exercise.
• Undue Attention to the Selected Questions — When a question is "outted" as being below average, it's easy enough to fix. That's a desirable side effect but it will skew the evaluation towards "hand-crafted", manipulated questions.
• Meta Not Always Ideal for Crowdsourcing — In all likelihood, a few users are going to do the majority of the work. For communities with less meta participation, this may not happen at all. There may be cases where round table-style "chat event" is warranted.

Answer 3

Here's a possible approach:

• SE staff generates a list of ten randomly chosen main questions and either posts the questions below themselves or provides the list to a community moderator, and let the mod post the questions and administer the process.

• For each evaluation round, post ten meta questions.

• Each meta question should contain a link to a randomly-chosen main question.

• Consider locking the chosen main questions for the duration of the evaluation, to prevent the evaluation from incorporating improvements that come as a result of the evaluation.

• Each meta question should also contain an explanation of the process (Why and How), including clear guidelines for the criteria that define "Better," "Par," and "Worse," and what sorts of searches are expected (including searches for conceptual synonyms, variant spellings, etc.).

• Community members should be invited to post answers to the meta questions containing a list of searches they did, an evaluation of the relevant contents of each search compared to the Q&A provided, and a summary, concluding with an overall rating of the question. Alternatively, such answers could be pre-posted with blank templates as community wikis.

• Other community members should be invited to vote for analyses that they think followed the process well and to comment on or edit those that they think are incomplete or incorrect.

• Whoever administers the process (SE staff or community mod) should accept one answer per meta question, taking both community votes and their own judgement into account.

• Whoever administers the process should sum up the results of the process based on the accepted answers in a final meta post and then unlock the main questions if they were locked.

• Consider creating a badge for having answers to these special meta questions accepted, as an incentive for community members to participate in the process.

• To reduce the potential for site-partisan bias, perhaps assign an auditor who's active on other site metas but not this one to review and challenge all of the meta posts.

Answer 4

I have a slightly different take on this. I think presenting users with 10 questions to research and review at once (and doing it at regular intervals) is a good way to either get few evaluations or poor evaluations. We have a new and improved Review section on every site, and I propose that we integrate quality evaluation into it.

The idea

• There's a new tab in Review called, say, Quality Evaluation. It can be unlocked once a user hits a certain reputation threshold similar to the Suggested Edits tab. This increases the odds (though doesn't guarantee) that the user has enough experience on the site/in the site's domain to be able to judge the responses.
• The Quality Evaluation Tab contains an ongoing list of 10 questions. The user has the option to click "Review question" which will expand the question (and answers? - might be too much information for an inline review) and show the following:
  • A dropdown list for the overall grade for the answers compared to the rest of the internet: better, par, worse.
  • A textbox for comments explaining the rating given above with some description encouraging the user to provide links to answers on other sites, etc.
  • A Google search URL with the question's title pre-populated in it to make it easier for the reviewer to start on researching answers on other sites. Ideally the question title would lead to relevant search results.
• Once 10 people (or some other reasonable number... perhaps scaled by the size of the site?) review a question, it disappears from the list and another question comes in to take its place.
• Questions are chosen by some heuristic (open for X amount of time, not closed, non-negative score, etc.)
• Could show a rolling stat summary in the sidebar along the lines of "out of 30 recently reviewed questions, 15 had answers better than the rest of the internet".

Questions

• What happens once a question and its answers are reviewed? A monthly summary of ratings posted to meta? Just something available for mods to view over time?
• How do we filter out junk/abusive reviews? Run each through 10k users/mods? Throw proposed reviews into suggested edits?

Pros

• Nobody has to review 10 questions at once. I think it's more important to have one quality review than 10 average reviews.
• Mostly automatic.
• Doesn't clutter meta.
• Ongoing review should turn up a more representative sampling of questions.
• Could be tied into the badge system.
• Could be easily done on graduated sites as well to make sure they stay on track.

Cons

• Ongoing review might be too much: too much work, too much information. This could be mitigated by manual selection of questions once a month or at some other interval.
• Peer-review of edits is difficult and might requite a more cumbersome approach than doing it all on meta.
• Not sure how to display/analyze the results. Lower visibility of review results to the community.

Answer 5

Given 10 questions one would need the following as a separate check

• Votes
  • If possible a breakdown of votes by a reputation bracket (though a shallow indicator, it would be good to know how large is the portion of top users watch this question)
  • Vote velocity of each user (Does the user vote 30-40 times per day?)
• Feedback Votes
• Activity of the tag itself (how popular is the tag overall? recently?)
• Search keywords that lead to the question
• Views over the age of the question

Now to evaluations, we need three-five outside reviewers and two-three in house.

---

Unbiased selection of sources

The order in which they review sources should change.

For example given 4 sources for question 1,

The order should be something like

• Source 1
• Source 3 (SE site)
• Source 4
• Source 2

And then for the next question for a next 4 sources for question 2

• Source 2
• Source 4 (SE site)
• Source 1
• Source 3

The reason is that we don't want one source giving influence to how other sources are seen.

There also needs to a base level question. A question that we know for sure is indeed high quality regardless of the source. Then we match this base review against all reviews.

Then the next section is the actual evaluation of the source,

---

Types of answers

1. There are detailed quality answers which explain to the user the solution in full no mater the level of the user such that no outside sources are needed to decipher
2. Concise answers with links to extended pages of documentation. These are answers which answer the OP and are for all intents and purposes correct. But for a next user would require deciphering via the linked documentation.
3. Concise answer with bias. "You should use X, because I use it at my company and it works great"
4. A link to the documentation/help resource page.

This list is not complete, there are many more types of correct answers

Assuming every answer one reviews is correct we basically need a list of all the different types of answers and then have the reviewer assign his opinion into one of the these categories.

---

Readability

We also next a section for readability. This should be different from the correctness of the answer. How easy to digest is the answer? Would the addition of a diagram help? Would the addition of LaTeX assist? Is the image distracting from the answer? (XKCD I am looking at you). Basically how quickly is the user able to collect the answer and go on his/her way.