Five months ago, Teresa Dietrich posted an introduction and a mission statement of sorts, laying out some admissions of past failures, and intentions for the new year. This was followed by two Roadmaps: 1st quarter and 2nd quarter.
At the outset, I summarized my impressions as:
I like nice words. I'm a big believer in nice words. I try to write nice words... But it is actions that tell you whether nice words are sincere.
Lots of other folks expressed similar sentiments; we all want to believe that change is possible, but... We've heard it all before; we wanna see change.
2019 was a rough year for Stack Exchange, Inc. D.B.A. Stack Overflow... Particularly the second half. I think it's fair to say that if the company had been assigned a letter grade for its work in that half of the year, it would have been a solid F. Not that nothing good happened... But, it's hard to keep track of all that's happening, especially when things are chaotic and a lot of things are being done badly. It's all too easy to remember the last action, for good or ill, and lose sight of the larger picture.
The other evening, I was chatting with some other folks and we had the idea of a "scorecard" or "report card" - something akin to the little grade summaries kids used to bring home from school in The Before Times. Something that we could use to encourage ourselves to take a more holistic view of the company's activity.
So, I put one together.
I weighed three factors when determining a grade for each individual task:
- Completion: Was the task actually performed? Note that I special-cased tasks that I could find no evidence were ever attempted; I don't have perfect visibility into everything going on and it isn't fair to grade things I just missed entirely. Also, with COVID-19 being what it is, it seems wrong to expect everything to get done as planned. Such tasks are graded "I", and don't affect the overall grade average.
- Quality: Was the task done well - specifically, does it seem likely to have the intended effect?
- Reception: Was the task well-received? Poor presentation can ruin the finest meal, and many a well-done effort has been torpedoed by a tone-deaf rollout.
I blended the results of these three factors together, added a healthy jigger of Jack Daniel's, and serve the results out below:
This is the average of all the individual task grades below, on a scale of F=0, D=1, C=2, B=3, A=4.
Not A-Awesome, but... Slightly better than just barely adequate. A significant turnaround from last year!
Read on for details on how I evaluated individual tasks...
Per task grades
Site Satisfaction Surveys
I think there were supposed to be two of these reports, one for each quarter; I found one for March. There was a brief mention of the survey in May, but not much in the way of actual results.
I've written about the first survey post already; there was certainly some room for improvement in the second round. I haven't seen much in the way of follow-through from the first results though, leading me to suspect this task has been either dropped or heavily deemphasized.
I gotta fail this one simply because there's no obvious reason for it to be incomplete and ineffective; if the company is gonna make something like this a tentpole of its strategy, it should at least make mention of it when that pole is cut down. If this was simply a failure to follow through, then my comments under "Meta & Mod Commitments" apply.
This was slightly late, but... Covid. Beyond that, it was a welcome sight indeed - reception was good both here and in far-flung corners of the 'Net, and the company was responsive to feedback. Nice Job!
Community Feedback Framework/Matrix
It was scheduled, and it arrived. It's... Fine I guess? Reception was cool but not overtly negative. When asked about it at the time, I believe I compared it to a kitchen full of fancy appliances; looks good in photos, but doesn't tell you much about how the food will taste. See previous notes on the Site Satisfaction Survey - if you're not reporting & acting on your observations, having the tools/process doesn't accomplish much.
The Loop monthly blog posts
There does appear to have been one of these every month, even if the last one is missing the tag.
Meta & Mod Commitments
This was published and very well received, with timely follow-ups.
So, Completion and Reception get top marks. Nice job!
...But, uh, quality... Here I have to debate the premise a bit. See, this task fell under the category of "Engaging with Community"; of the 11 tasks in that category, this one was by far the most appropriate: let's face it, updating legal agreements and publishing blog posts are... Important tasks, but not particularly engaging ones. This task is the one with the most potential to actually fulfill the promise Teresa made back in February:
We (the Community, Product and Engineering teams) will endeavor to clarify and reset when necessary what you can expect from the company. We will also provide as much context as we can for policy, decisions, and actions that we take within legal and regulatory constraints - we want you to understand why we’re making changes, not just that we’re making them.
That's a lot of weight for one task to bear. But, it must: there is no other task that can. And so when evaluating the quality of the efforts made here, I must keep that promise firmly in mind.
Let's step back for a sec & unpack the problem here: the core premise of this process is that issues will be reported by users, triaged by a combination of volunteers (moderators) and employees, then reviewed and resolved by employees of the company. There are two potential flaws in this:
- Volunteers may not have the time, experience or inclination to triage the full volume of incoming issues, even those of a time-sensitive nature.
- Employees may not have the time, experience or inclination to do so either.
There are a couple of common ways to mitigate these flaws:
Hire an actual quality assurance team. Folks to identify and triage defects, talk to users, shepherd issues through the bureaucracy to resolution. Once upon a time, this fell among the Community Manager duties - but, the company is running on a skeleton crew of CMs at this point, and if they can't even flesh out that team then hiring a team of QA engineers is unlikely. Further reading: Top Five (Wrong) Reasons You Don't Have Testers (lots of good points, and some that haven't aged well; pay for good software quality engineers FFS!)
Guide & coordinate the volunteers. This needs a bit of explanation, for folks not familiar with large volunteer-driven projects (such as open source software): the basic idea is, you recruit a bunch of folks who want to be involved in the development process (not just people with a patch) and you set them up to do various parts of the process: issue triage, testing, documentation, etc. Given the obvious budget constraints, this is probably the only option open to SO, Inc.
Crucially, #2 doesn't just rely on volunteers randomly deciding to do stuff; it makes them an integral part of the development process. I currently work with folks who've been doing this as part of the PostgreSQL project for many, many years - it is demonstrably an approach that allows a project to stay healthy long-term.
And this is where I gotta ding this task on Quality: it aims too low. The current published commitment aims for 50% resolution to tagged questions; that makes even stuff triaged as "needs a response" a coin-flip, and leaves the vast bulk of issues out in the cold. Critically, it doesn't attempt to formally organize or guide volunteers, much less involve them in the process.
This is a problem for the express goal of the task, but a bigger problem for the future of the company itself. This software isn't just a critical bit of infrastructure for programmers and other professionals; it's also - in the form of Teams - what the company itself is banking on to provide for its future. Meta Stack Overflow is the official support venue for this software, and beyond that all issues on these sites are the public evidence of the quality - or lack thereof - that potential customers can expect. And... Customers could be forgiven for having low expectations right now.
I'll provide two examples to illustrate what I'm talking about:
This feature was rolled out recently, with apparently no advance preparation done for the volunteers who would be responsible for managing bugs and feedback. They were blindsided; one of them responded immediately by creating a script to disable the feature! This isn't engaging with moderators & meta, it's actively fighting them!
To illustrate just how damaging this can be, consider this bug which the Reactions feature introduced: it is actively skewing the data being collected for the experiment. In spite of a non-trivial amount of attention from users and its connection to a currently-active experiment, it received no status tag for over two weeks, and remains unfixed - as a result, additional effort will be needed to correct the data being collected before any conclusions can be drawn from it. It should have been immediately flagged as high-priority, corrected, and the erroneous data purged - but lacking employee bandwidth and volunteer guidance, it was overlooked. Consequently, bad data continued to be recorded through the end of the test.
This request was posted as a direct result of the Site Survey results (discussed above). No status tag was added for two months; after tagging, it got an employee comment promising an answer... But that was over a month ago, and no answer is to be found.
I happen to be extremely familiar with both the behavior of the feature being discussed and the logic that underpins it; I was part of the team that designed the feature originally, part of the team that revamped it a few years later, the person who maintained the public documentation for it for years, and most recently the person to fix bugs reported in it. So, trust me when I assure you, this is not a difficult issue to address. Yet, in spite of a substantial amount of user feedback, in spite of volunteer efforts to adhere to the documented process, it has no resolution.
In both cases, we can see that the current approach stands to cripple the foundational quality of the system, with defects spidering out to affect everything else in this list. There is an envious amount of data collected regarding these reports automatically, without anyone putting in any manual work; if the company is not using it, then they are setting themselves up for destruction. There is an envious amount of volunteer labor & expertise ready at hand to assist in these efforts; by locking volunteers out of the development process, the company squanders their already-limited resources.
Quality & communication ain't optional; they're not the sprinkles on the donut, they're the whole donut. Execution here is gonna make or break the company's future, and they cannot continue to approach it with half measures and hope to survive.
This happened; reception was cool, and there was no real engagement beyond the announcement so far as I can tell, with concerns left unaddressed. I'm... Being pretty generous with a C here, but I also don't care that much: the biggest privacy issues in recent memory arguably violated numerous policies and were in no way slowed by the existence of such policies; they're a necessary bit of theater. If you need to take your privacy seriously, please don't rely on policies.
Internal Community Monthly Newsletter
No idea if this happened or if it did anything useful.
In 2019, the company was in the throes of a serious addiction, one that very nearly destroyed it. The completion of this task is evidence that, while they aren't entirely free, they may finally be on the long road to recovery.
I know a lot of former meth addicts. It's hard for me to empathize with them, to commiserate in the pride that they feel after a month, a year, a decade of abstaining; I've managed to make it my whole life without indulging in recreational amphetamines, how hard can it be? But... When I hear their stories, hear what they've come back from - lost friends, families, children, health, freedom... I can start to appreciate just how much of an accomplishment it is for them to be where they are.
Stack Overflow, Inc.'s addiction wasn't meth; it was a far more insidious and far more destructive vice: pride.
When I saw the first draft of the v2 moderator agreement back at the start of January, I figured that was rock-bottom for Stack Overflow. It was overtly insulting through and through, and possibly even stood to put moderators in a position of violating contracts they'd signed with their actual employers. It seemed almost engineered to force a good chunk of the rest of the moderators to quit, to finally destroy a decade of work and salt the earth on which it had stood. The company's addiction had driven them to the brink of utter disaster, yet, even as former colleagues fled and the community raged and pleaded with them, even as some loyal employees tried to stage the professional equivalent of an intervention... They seemed incapable of letting go of the hubris that was destroying the organization from within.
The final draft that was released isn't perfect; it isn't even particularly good. But, it is sufficient. And given where it started, I have to respect the accomplishment. It took half a year of struggling - and like most recovering addicts the company will never be entirely free from that struggle - but, they were able to persevere and release it in a state that didn't destroy everything. Reception was as good as can be expected given the circumstances, and engagement has been solid.
New Question Asking UX: Int'l
This appears to have happened, along with all other sites. A long time in coming, and much welcomed!
New Close UX
There has actually been some significant work done here - but I'm gonna be generous and give it an (I)ncomplete grade, because this is all just foreplay: the real make-or-break changes are yet to come. Specifically:
- Hiding closed questions
- Edit-triggered reopening
Those changes stand to fundamentally alter the nature of closing on SO & SE, and affect a tremendous number of people who don't even have accounts on the network - the company is gonna have to bring its A-game to pull it off, and hopefully they do just that. To date, reception for this project has been decidedly mixed - they've gotten some accolades but also a LOT of criticism, and frankly they're kinda slacking when it comes to selling the core philosophy of this project. But, they can still turn it around, if they really really want to. See persistently relevant notes under the "Meta & Mod Commitments" task.
Email Drip Campaign
What can I say that hasn't already been said? This was an abject failure. It takes a lot to embarass me, and yet I can't help but feel embarrassed on behalf of the company!
I've written a lot of insulting, scary messages to folks in my day; heck, there are still moderators who hate my guts because I chided them over something in such a tone. I've even written scary, insulting copy (some of which is still part of the SO UI, affecting almost every active user of certain features to this day). And of course, I've screwed up when sending emails on too many occasions to count. But I've never managed the trifecta: sending scary, insulting copy to a million+ users. That'd be impressive, if the goal had been "stop everyone who contributes to this site in their tracks" - but, uh, it was actually "increase user engagement". I guess someone thought it was "opposite day"? Ouch.
The one bright spot in this task is that it was a drip campaign and they stopped it early. Which is good for me, because if they'd actually managed to drive away every single contributor I think I would have needed a grade lower than "F".
Been wanting this forever, along with loads of other people. This is one of those features that's been requested repeatedly since the first days of the site - heck, I remember folks building 3rd-party tools to do this before there was even an API!
Needless to say, it's great to see it land finally. I could be petty and wish for fine-grained notification controls, but this is by itself a huge step forward. Great Work!
Convene Pro Tempore Moderator Council
This was done, and reception was excellent! However, the role given to the new Council is... Disappointing.
As I noted in "Meta & Mod Commitments", it's not enough to have folks volunteering - you gotta organize them, give them stuff to do. If the Council is to be an integral part of this operation and not just a scapegoat, the company must provide - and the council must demand - actual responsibility, authority, and support. I recognize that many of the current members do not want that responsibility; faced with a situation in which the company is determined to shoot itself in the foot again, they'd prefer not to stand in the line of fire... That's entirely understandable. But also... Useless. If the company wishes to keep what is left of its feet, it must entrust its bullets to someone else.
It, uh, happened. I saw some folks active at the outset, hopefully there were a lot more behind the scenes. Frankly, just the effort to do it means a lot, so I'll call it a success purely for image.
Some of my associates disputed this generous grading, asserting that a lack of visibility here is a failure in and of itself - I strongly disagree. As with the Moderator Agreement, it is important to remember the depths from which the company is attempting to ascend: not so long ago, suggestions that employees should participate on the sites which represent their employer's most well-known creation were met with derision and overt hostility; therefore, seeing this done at all is a large step forward.
Moderator Reinstatement update
It happened. I think it's a slight improvement, but there's no denying that the reception was bad and overall the effort likely fell far short of its goals. I've written a lot about this elsewhere, but I don't think it's that interesting so I won't reiterate it here; George summed it up best, I think: "This process is fundamentally flawed because the moderator removal process is fundamentally flawed."
I think we have to consider the analogy I used under "Moderator Agreement" here: the company is fighting an addiction, and... There are good days, and not-so-good days.
No info on this; maybe it happened? But I haven't seen any updates to the process on meta, so I'm gonna assume it's still in planning.
CC License update update
This happened. It took far, faaaaaaar too long, and the kinks will probably take a lot longer to fully resolve... But a response finally managed to tackle the concerns head-on, work to shore up defects has been prompt and ongoing, and the reception has been good. Thumbsup!
Full disclosure: in my heart, I want to give this an A for effort. But my associates advised me to be realistic; this came two years after the license was changed, and followed vast amounts of misleading or belittling responses to concerns. As much as I appreciate the solution, there's no denying that it is a solution to a problem that the company itself created and nurtured; awarding extra points in this scenario would be foolhardy.
Moderator Training Launch
My associates advise me that this was begun, and is.... Adequate, if not particularly actionable. Hopefully that improves as work progresses.
Moderator Town Halls
Maybe this happened and mods are just super quiet about it? Donno, can't grade.
Collect New User Feedback
I really expected to hear something about this, but either I missed it or it didn't happen.
This happened... On SO proper. It's really nice! Now just need the other 300-some sites & meta sites to get it.
Review queue phase 2 updates
Discovery began in April, haven't heard much since. Got high hopes for it, but... See my comments in the Meta & Mod Commitments above: they gotta get better about talking through this stuff; "ivory tower" development doesn't fly. To get this designed and built out successfully, the company will need to enlist volunteers in the development process, and soon.
This was a fun exercise, and it really got me to think about everything that's been done so far this year. There was a lot that I'd forgotten about, and it was informative to revisit them all!
Big thanks to the current and former SO/SE users who assisted me on this little project - your input was invaluable, and if any of you wish to take credit or shoulder blame I'm more'n happy to share both. Thanks also to Jack Daniel's Tennessee Whiskey, for inspiration and motivation.
I think the grades above are pretty fair, but... Don't trust my judgement - try this exercise for yourself! I've tried to link to relevant material for each task that I graded; have a look, & post your own grades below.