Regarding the need for persistence of the New Contributor Indicator, Animuson pointed out, there are some particular "users who definitely should be seeing it" (Emphasis in original).
I think we can all agree with Animuson that there are some particular users who are being rude, unkind or otherwise unwelcoming. In fact, about 50 of the 57 people who dedicated a decent portion of time to review and classify nearly 250 comments each found at least one unwelcoming or rude comment in the batch of comments they saw, as reported in this diagram:
Make sure to pay attention to the axis, the 6 or 7 unit high blue bar on the far right indicate the people who thought all of the comments they evaluated were fine, the far left indicates that approximately 51 people found zero rude or abusive comments in their sample, the orange spread is people's opinions on the amount of unwelcoming comments they saw, and from there you can probably be sure you're understanding the rest of the graph correctly.
Now, there have been plenty of discussions about what kinds of comments are actually unwelcoming and what kinds are being blown out of proportion, and I'm not proposing a debate on that in this feature request, nor a discussion about the validity of that survey. All of these things have been discussed to death. Let's just assume that we've got a few bad apples who are intentionally being unwelcoming, and maybe a few of us are careless from time to time and our comments get poorly received.
Back in June, some of you might remember that on the Stack Overflow Blog, the corporation bragged about having a Snarky or Unwelcoming Comments Classification App.
With all this information available to us, I think we finally have the right tools with which to evaluate how effective this new New Contributor Indicator actually is.
Proposal for How we could evaluate the effectiveness of the New Contributor Indicator
Step one: Get a sample of users and their comments, grouping users into categories based on the quality of their comments. The comments can be evaluated through use of the Unwelcoming and Snark Comment Detection App (Okay, I'm just going to call it the Snark Detector, that's too much of a mouthful).
This will need to be done with care, and obviously these results won't be made publicly available. My idea of how to classify users is this:
Using data from the last 5-ish months, since the original unwelcoming blog was posted, up until the day the New Contributor Indicator was revealed, classify each user's comments over each month. It's important that we get a sample from this timeframe, as many users have made at least small changes to their behavior due to the large impact this welcoming phase has had. Comments older than that will skew the results.
Also important is to group these results month by month. Nearly every month there have been updates which might have had effects on people's commenting habits, so it's important for this test to separate out users who have commented with the same amount of Snark/Unwelcomingness in spite of the updates from those whose comment quality has increased (or hopefully we won't see a decline in quality from anyone, that is, someone who has become more rough with their words in the last months, but if it happens, we'll classify those people as well).
I'm hoping for at least a few hundred users and their comments will be surveyed, which shouldn't be too tough since it's an automated classifier.
Step two: Classify the different types of users in this sample based on their normal use of comments and how unwelcoming they are.
Some of the different categories I can imagine would be important are going by quantity of comments (We should group people who make less than 10 comments a month a little different than someone who makes more than 100 comments a month), then by a certain amount of how unwelcoming they are (Some users never post anything remotely unwelcoming, others post some flat out abusive things, some are generally unwelcoming, so let's break them up into groups), and within those groups, make another division between users who have changed their commenting patterns during these last few months and those who have stayed constant. Obviously viewing the data there might be more meaningful ways to split them up, but the point is that we want to isolate certain types of people enough so that when we look at how the New Contributor Indicator's presence is affecting them, we can know if it's having any decent effect.
Step three: After a period of time (Maybe 2 months for an initial report, 6 months for a more complete report), evaluate this sample of users' comments again.
We want to know something soon, since the indicator is rather invasive and we want to know if it's working or not. We really can't hope to be able to see any change for less than a couple months of data though. And even if we do draw some initial conclusions after just a couple or a few months of data, if there's enough data to indicate that it's a useful study, let's keep it going for longer. We want to really understand if this is correcting the unhelpful behavior any.
Step four: Make a publicly (and very anonymous) report available of any trends that the data results in.
My hope would be that we can see a meaningful decrease in the unwelcoming-ness of any users who were previously rated as having a significant rate of unwelcoming comments per month. How we want to define meaningful and significant will have to be up to the people who can actually see the data. On the other hand, we may see relatively no change across the board, or even an increase. The point is, if this survey of users demonstrates an actionable change in behavior based on the introduction of the New Contributor Indicator, we can justify its continued existence. If it does not, then we can hopefully honor the requests to hide the indicator, or just get rid of it completely.
There are users who definitely should be seeing the New Contributor Indicator, and hopefully it will help them to be more welcoming all across the network. There are also users who feel that the New Contributor Indicator won't help anything at all. This feature request is simply to evaluate how welcoming people are being using the Comments Classification App, and seeing if the New Contributor Indicator is having any effect on that, and I've given my suggestion for one way this can be implemented and studied.
I welcome answers that propose improvements to this plan or alternative ways to study the effectiveness of the New Contributor Indicator. And with some luck and a little time, we can either get some good results out of the New Contributor Indicator that help users to be more welcoming, or we find out that we can get rid of the indicator all together.
Subnote: As per Yvette's request that we don't make "snipes and attacks at the site's new policies," I've tried to make this as neutral as I could, but you might notice my distaste for the entire Welcoming phase that we're going through in my wording here and there in places that it's slipped by me. If you do notice me being overly negative towards the site's policies, please comment with the phrasing I should fix, as I want this to be a useful feature request, not just another rant hidden behind the guise of a request.