106

Five months ago, Teresa Dietrich posted an introduction and a mission statement of sorts, laying out some admissions of past failures, and intentions for the new year. This was followed by two Roadmaps: 1st quarter and 2nd quarter.

At the outset, I summarized my impressions as:

I like nice words. I'm a big believer in nice words. I try to write nice words... But it is actions that tell you whether nice words are sincere.

Lots of other folks expressed similar sentiments; we all want to believe that change is possible, but... We've heard it all before; we wanna see change.

2019 was a rough year for Stack Exchange, Inc. D.B.A. Stack Overflow... Particularly the second half. I think it's fair to say that if the company had been assigned a letter grade for its work in that half of the year, it would have been a solid F. Not that nothing good happened... But, it's hard to keep track of all that's happening, especially when things are chaotic and a lot of things are being done badly. It's all too easy to remember the last action, for good or ill, and lose sight of the larger picture.

The other evening, I was chatting with some other folks and we had the idea of a "scorecard" or "report card" - something akin to the little grade summaries kids used to bring home from school in The Before Times. Something that we could use to encourage ourselves to take a more holistic view of the company's activity.

So, I put one together.

Methodology

I weighed three factors when determining a grade for each individual task:

  1. Completion: Was the task actually performed? Note that I special-cased tasks that I could find no evidence were ever attempted; I don't have perfect visibility into everything going on and it isn't fair to grade things I just missed entirely. Also, with COVID-19 being what it is, it seems wrong to expect everything to get done as planned. Such tasks are graded "I", and don't affect the overall grade average.
  2. Quality: Was the task done well - specifically, does it seem likely to have the intended effect?
  3. Reception: Was the task well-received? Poor presentation can ruin the finest meal, and many a well-done effort has been torpedoed by a tone-deaf rollout.

I blended the results of these three factors together, added a healthy jigger of Jack Daniel's, and serve the results out below:

Overall grade

C+

This is the average of all the individual task grades below, on a scale of F=0, D=1, C=2, B=3, A=4.

Not A-Awesome, but... Slightly better than just barely adequate. A significant turnaround from last year!

Read on for details on how I evaluated individual tasks...

Per task grades

Site Satisfaction Surveys

F

I think there were supposed to be two of these reports, one for each quarter; I found one for March. There was a brief mention of the survey in May, but not much in the way of actual results.

I've written about the first survey post already; there was certainly some room for improvement in the second round. I haven't seen much in the way of follow-through from the first results though, leading me to suspect this task has been either dropped or heavily deemphasized.

I gotta fail this one simply because there's no obvious reason for it to be incomplete and ineffective; if the company is gonna make something like this a tentpole of its strategy, it should at least make mention of it when that pole is cut down. If this was simply a failure to follow through, then my comments under "Meta & Mod Commitments" apply.

Developer Survey

A

This was slightly late, but... Covid. Beyond that, it was a welcome sight indeed - reception was good both here and in far-flung corners of the 'Net, and the company was responsive to feedback. Nice Job!

Community Feedback Framework/Matrix

C

It was scheduled, and it arrived. It's... Fine I guess? Reception was cool but not overtly negative. When asked about it at the time, I believe I compared it to a kitchen full of fancy appliances; looks good in photos, but doesn't tell you much about how the food will taste. See previous notes on the Site Satisfaction Survey - if you're not reporting & acting on your observations, having the tools/process doesn't accomplish much.

The Loop monthly blog posts

C

There does appear to have been one of these every month, even if the last one is missing the tag.

Meta & Mod Commitments

C

This was published and very well received, with timely follow-ups.

So, Completion and Reception get top marks. Nice job!

...But, uh, quality... Here I have to debate the premise a bit. See, this task fell under the category of "Engaging with Community"; of the 11 tasks in that category, this one was by far the most appropriate: let's face it, updating legal agreements and publishing blog posts are... Important tasks, but not particularly engaging ones. This task is the one with the most potential to actually fulfill the promise Teresa made back in February:

We (the Community, Product and Engineering teams) will endeavor to clarify and reset when necessary what you can expect from the company. We will also provide as much context as we can for policy, decisions, and actions that we take within legal and regulatory constraints - we want you to understand why we’re making changes, not just that we’re making them.

That's a lot of weight for one task to bear. But, it must: there is no other task that can. And so when evaluating the quality of the efforts made here, I must keep that promise firmly in mind.

Let's step back for a sec & unpack the problem here: the core premise of this process is that issues will be reported by users, triaged by a combination of volunteers (moderators) and employees, then reviewed and resolved by employees of the company. There are two potential flaws in this:

  1. Volunteers may not have the time, experience or inclination to triage the full volume of incoming issues, even those of a time-sensitive nature.
  2. Employees may not have the time, experience or inclination to do so either.

There are a couple of common ways to mitigate these flaws:

  1. Hire an actual quality assurance team. Folks to identify and triage defects, talk to users, shepherd issues through the bureaucracy to resolution. Once upon a time, this fell among the Community Manager duties - but, the company is running on a skeleton crew of CMs at this point, and if they can't even flesh out that team then hiring a team of QA engineers is unlikely. Further reading: Top Five (Wrong) Reasons You Don't Have Testers (lots of good points, and some that haven't aged well; pay for good software quality engineers FFS!)

  2. Guide & coordinate the volunteers. This needs a bit of explanation, for folks not familiar with large volunteer-driven projects (such as open source software): the basic idea is, you recruit a bunch of folks who want to be involved in the development process (not just people with a patch) and you set them up to do various parts of the process: issue triage, testing, documentation, etc. Given the obvious budget constraints, this is probably the only option open to SO, Inc.

Crucially, #2 doesn't just rely on volunteers randomly deciding to do stuff; it makes them an integral part of the development process. I currently work with folks who've been doing this as part of the PostgreSQL project for many, many years - it is demonstrably an approach that allows a project to stay healthy long-term.

And this is where I gotta ding this task on Quality: it aims too low. The current published commitment aims for 50% resolution to tagged questions; that makes even stuff triaged as "needs a response" a coin-flip, and leaves the vast bulk of issues out in the cold. Critically, it doesn't attempt to formally organize or guide volunteers, much less involve them in the process.

This is a problem for the express goal of the task, but a bigger problem for the future of the company itself. This software isn't just a critical bit of infrastructure for programmers and other professionals; it's also - in the form of Teams - what the company itself is banking on to provide for its future. Meta Stack Overflow is the official support venue for this software, and beyond that all issues on these sites are the public evidence of the quality - or lack thereof - that potential customers can expect. And... Customers could be forgiven for having low expectations right now.

I'll provide two examples to illustrate what I'm talking about:

  1. This feature was rolled out recently, with apparently no advance preparation done for the volunteers who would be responsible for managing bugs and feedback. They were blindsided; one of them responded immediately by creating a script to disable the feature! This isn't engaging with moderators & meta, it's actively fighting them!

    To illustrate just how damaging this can be, consider this bug which the Reactions feature introduced: it is actively skewing the data being collected for the experiment. In spite of a non-trivial amount of attention from users and its connection to a currently-active experiment, it received no status tag for over two weeks, and remains unfixed - as a result, additional effort will be needed to correct the data being collected before any conclusions can be drawn from it. It should have been immediately flagged as high-priority, corrected, and the erroneous data purged - but lacking employee bandwidth and volunteer guidance, it was overlooked. Consequently, bad data continued to be recorded through the end of the test.

  2. This request was posted as a direct result of the Site Survey results (discussed above). No status tag was added for two months; after tagging, it got an employee comment promising an answer... But that was over a month ago, and no answer is to be found.

    I happen to be extremely familiar with both the behavior of the feature being discussed and the logic that underpins it; I was part of the team that designed the feature originally, part of the team that revamped it a few years later, the person who maintained the public documentation for it for years, and most recently the person to fix bugs reported in it. So, trust me when I assure you, this is not a difficult issue to address. Yet, in spite of a substantial amount of user feedback, in spite of volunteer efforts to adhere to the documented process, it has no resolution.

In both cases, we can see that the current approach stands to cripple the foundational quality of the system, with defects spidering out to affect everything else in this list. There is an envious amount of data collected regarding these reports automatically, without anyone putting in any manual work; if the company is not using it, then they are setting themselves up for destruction. There is an envious amount of volunteer labor & expertise ready at hand to assist in these efforts; by locking volunteers out of the development process, the company squanders their already-limited resources.

Quality & communication ain't optional; they're not the sprinkles on the donut, they're the whole donut. Execution here is gonna make or break the company's future, and they cannot continue to approach it with half measures and hope to survive.

Privacy Policy Update

C

This happened; reception was cool, and there was no real engagement beyond the announcement so far as I can tell, with concerns left unaddressed. I'm... Being pretty generous with a C here, but I also don't care that much: the biggest privacy issues in recent memory arguably violated numerous policies and were in no way slowed by the existence of such policies; they're a necessary bit of theater. If you need to take your privacy seriously, please don't rely on policies.

Internal Community Monthly Newsletter

I

No idea if this happened or if it did anything useful.

Moderator Agreement

A

In 2019, the company was in the throes of a serious addiction, one that very nearly destroyed it. The completion of this task is evidence that, while they aren't entirely free, they may finally be on the long road to recovery.

I know a lot of former meth addicts. It's hard for me to empathize with them, to commiserate in the pride that they feel after a month, a year, a decade of abstaining; I've managed to make it my whole life without indulging in recreational amphetamines, how hard can it be? But... When I hear their stories, hear what they've come back from - lost friends, families, children, health, freedom... I can start to appreciate just how much of an accomplishment it is for them to be where they are.

Stack Overflow, Inc.'s addiction wasn't meth; it was a far more insidious and far more destructive vice: pride.

When I saw the first draft of the v2 moderator agreement back at the start of January, I figured that was rock-bottom for Stack Overflow. It was overtly insulting through and through, and possibly even stood to put moderators in a position of violating contracts they'd signed with their actual employers. It seemed almost engineered to force a good chunk of the rest of the moderators to quit, to finally destroy a decade of work and salt the earth on which it had stood. The company's addiction had driven them to the brink of utter disaster, yet, even as former colleagues fled and the community raged and pleaded with them, even as some loyal employees tried to stage the professional equivalent of an intervention... They seemed incapable of letting go of the hubris that was destroying the organization from within.

The final draft that was released isn't perfect; it isn't even particularly good. But, it is sufficient. And given where it started, I have to respect the accomplishment. It took half a year of struggling - and like most recovering addicts the company will never be entirely free from that struggle - but, they were able to persevere and release it in a state that didn't destroy everything. Reception was as good as can be expected given the circumstances, and engagement has been solid.

New Question Asking UX: Int'l

A

This appears to have happened, along with all other sites. A long time in coming, and much welcomed!

New Close UX

I

There has actually been some significant work done here - but I'm gonna be generous and give it an (I)ncomplete grade, because this is all just foreplay: the real make-or-break changes are yet to come. Specifically:

  • Hiding closed questions
  • Edit-triggered reopening

Those changes stand to fundamentally alter the nature of closing on SO & SE, and affect a tremendous number of people who don't even have accounts on the network - the company is gonna have to bring its A-game to pull it off, and hopefully they do just that. To date, reception for this project has been decidedly mixed - they've gotten some accolades but also a LOT of criticism, and frankly they're kinda slacking when it comes to selling the core philosophy of this project. But, they can still turn it around, if they really really want to. See persistently relevant notes under the "Meta & Mod Commitments" task.

Email Drip Campaign

F

What can I say that hasn't already been said? This was an abject failure. It takes a lot to embarass me, and yet I can't help but feel embarrassed on behalf of the company!

I've written a lot of insulting, scary messages to folks in my day; heck, there are still moderators who hate my guts because I chided them over something in such a tone. I've even written scary, insulting copy (some of which is still part of the SO UI, affecting almost every active user of certain features to this day). And of course, I've screwed up when sending emails on too many occasions to count. But I've never managed the trifecta: sending scary, insulting copy to a million+ users. That'd be impressive, if the goal had been "stop everyone who contributes to this site in their tracks" - but, uh, it was actually "increase user engagement". I guess someone thought it was "opposite day"? Ouch.

The one bright spot in this task is that it was a drip campaign and they stopped it early. Which is good for me, because if they'd actually managed to drive away every single contributor I think I would have needed a grade lower than "F".

Follow Questions

A

Been wanting this forever, along with loads of other people. This is one of those features that's been requested repeatedly since the first days of the site - heck, I remember folks building 3rd-party tools to do this before there was even an API!

Needless to say, it's great to see it land finally. I could be petty and wish for fine-grained notification controls, but this is by itself a huge step forward. Great Work!

Convene Pro Tempore Moderator Council

C

This was done, and reception was excellent! However, the role given to the new Council is... Disappointing.

As I noted in "Meta & Mod Commitments", it's not enough to have folks volunteering - you gotta organize them, give them stuff to do. If the Council is to be an integral part of this operation and not just a scapegoat, the company must provide - and the council must demand - actual responsibility, authority, and support. I recognize that many of the current members do not want that responsibility; faced with a situation in which the company is determined to shoot itself in the foot again, they'd prefer not to stand in the line of fire... That's entirely understandable. But also... Useless. If the company wishes to keep what is left of its feet, it must entrust its bullets to someone else.

Community-Athon

C

It, uh, happened. I saw some folks active at the outset, hopefully there were a lot more behind the scenes. Frankly, just the effort to do it means a lot, so I'll call it a success purely for image.

Some of my associates disputed this generous grading, asserting that a lack of visibility here is a failure in and of itself - I strongly disagree. As with the Moderator Agreement, it is important to remember the depths from which the company is attempting to ascend: not so long ago, suggestions that employees should participate on the sites which represent their employer's most well-known creation were met with derision and overt hostility; therefore, seeing this done at all is a large step forward.

Moderator Reinstatement update

D

It happened. I think it's a slight improvement, but there's no denying that the reception was bad and overall the effort likely fell far short of its goals. I've written a lot about this elsewhere, but I don't think it's that interesting so I won't reiterate it here; George summed it up best, I think: "This process is fundamentally flawed because the moderator removal process is fundamentally flawed."

I think we have to consider the analogy I used under "Moderator Agreement" here: the company is fighting an addiction, and... There are good days, and not-so-good days.

Suicide Prevention

I

No info on this; maybe it happened? But I haven't seen any updates to the process on meta, so I'm gonna assume it's still in planning.

CC License update update

C

This happened. It took far, faaaaaaar too long, and the kinks will probably take a lot longer to fully resolve... But a response finally managed to tackle the concerns head-on, work to shore up defects has been prompt and ongoing, and the reception has been good. Thumbsup!

Full disclosure: in my heart, I want to give this an A for effort. But my associates advised me to be realistic; this came two years after the license was changed, and followed vast amounts of misleading or belittling responses to concerns. As much as I appreciate the solution, there's no denying that it is a solution to a problem that the company itself created and nurtured; awarding extra points in this scenario would be foolhardy.

Moderator Training Launch

C

My associates advise me that this was begun, and is.... Adequate, if not particularly actionable. Hopefully that improves as work progresses.

Moderator Town Halls

I

Maybe this happened and mods are just super quiet about it? Donno, can't grade.

Collect New User Feedback

I

I really expected to hear something about this, but either I missed it or it didn't happen.

Dark Mode

B

This happened... On SO proper. It's really nice! Now just need the other 300-some sites & meta sites to get it.

Review queue phase 2 updates

B

Discovery began in April, haven't heard much since. Got high hopes for it, but... See my comments in the Meta & Mod Commitments above: they gotta get better about talking through this stuff; "ivory tower" development doesn't fly. To get this designed and built out successfully, the company will need to enlist volunteers in the development process, and soon.

Final thoughts

This was a fun exercise, and it really got me to think about everything that's been done so far this year. There was a lot that I'd forgotten about, and it was informative to revisit them all!

Big thanks to the current and former SO/SE users who assisted me on this little project - your input was invaluable, and if any of you wish to take credit or shoulder blame I'm more'n happy to share both. Thanks also to Jack Daniel's Tennessee Whiskey, for inspiration and motivation.

I think the grades above are pretty fair, but... Don't trust my judgement - try this exercise for yourself! I've tried to link to relevant material for each task that I graded; have a look, & post your own grades below.

  • 37
    As someone who is still on the front lines daily, and has to deal with all of the fallout of mistakes and misjudgments made by the company, I think you're very over-generous in your grading. Even things that you rated highly, such as Dark Mode, were a disaster in practice. There appeared to be only one developer working on this, and the feature was shipped before it was even complete or tested. Large portions of the site simply weren't visible with Dark Mode enabled, including part of the election page. Bugs were reported and not addressed for over a month. Some aspects are still broken. – Cody Gray Jul 19 at 15:08
  • 7
    That's very fair, @Cody - I'm nothing if not a bright eyed optimist. I'd love to see which grades you (or the SO mod team as a whole) would assign to each task... – Shog9 Jul 19 at 15:10
  • 2
    This hangover brought to you by Jack – Shog9 Jul 19 at 15:31
  • 23
    I agree that the grading seems generous, if for no other reason than the comparison is to where the company was at the end of the year, instead of when communication started breaking down. Awarding points because the moderator agreement is better than the internal proposal seems like a gimme; better to compare it to the original version instead. Likewise for the rest of it; it would be better for a comparison against the start of the welcoming wagon, I feel, instead of the rock bottom start of the year. – fbueckert Jul 19 at 15:32
  • 2
    @Shog9 no mention to code of conduct. (As if it didn't need work, as if it weren't an issue. As if it was perfect as is....) – bad_coder Jul 19 at 15:41
  • 9
    To my knowledge, no work has been done on the CoC this year, @bad_coder, and crucially it wasn't on the published roadmap - if I were to expand this to encompass all the work yet to be done, I'd need a year and my liver would probably fail. – Shog9 Jul 19 at 15:44
  • 3
    Yeah, on the whole SEI has done far better than 2019 (which still isn't saying much) but some progress is better than none. Some meager praise is thusly deserved, lest we wind up with this response – Machavity Jul 19 at 16:33
  • 15
    @Shog9 The community, as well as the company whether it realizes it or not, is very fortunate to have you still participating here. I, at least, am very grateful for the time & effort you put into writing & posting this. As others have stated, I believe you were somewhat overly generous in some of your grading, but nonetheless this is one of the very best posts I've read among nearly all of the ones I've read on this site from over the past couple of years. Thank you for all of your work. I look forward to reading whatever additional excellent posts you grace us with in the future. – John Omielan Jul 19 at 19:49
  • 7
    @shog9 your grade on the moderator agreement was way too high. I would have given it a B-/C+ for not learning from their past mistakes. – George Stocker Jul 20 at 1:14
  • 6
    Now that I’ve read everything you’ve written on it (instead of scrolling down, seeing the grade, and mashing “comment”), I get why you gave it an A. If their draft was as bad as you say, then yes, this draft is much better and worthy of recognition. But that, of course, leaves the elephant of “why is this “not a terrible moderator agreement” where we want things? Shouldn’t we expect more out of SO INC? – George Stocker Jul 20 at 1:20
  • 3
    It probably won't surprise you to learn that, on about February, my expectations were extremely low, @george. I'm... Being cautious about how quickly I allow those to increase; plenty of other areas of life where I can get my fill of dashed expectations right now. Of course, that's just me. I heartily encourage y'all to try this exercise yourself, and share your results... – Shog9 Jul 20 at 1:33
  • 3
    It'd be nice if the Markdown here supported floating images, or even tables @trilarion – Shog9 Jul 20 at 13:03
  • 3
    This isn't a review of the 1H changelog, @gnat. I'm very happy with the switch to CommonMark, but... Weirdly enough, that wasn't on the roadmap! – Shog9 Jul 20 at 18:54
  • 4
    FWIW the CommonMark work was done by the Teams team. – Adam Lear Jul 20 at 22:49
  • 5
    Not on the Roadmap, but why is that so important when the change affected the entire network, and was not well received by many users? Some improvements to blockquotes 72 upvotes against 64 downvotes. – Mari-Lou A Jul 21 at 5:47
36

The grades are interesting, but I find the incompletes even more enlightening. Until someone put the results in one post, it was hard to visualize what's been done and what hasn't been started. It looks like theh report card of a student who took a full course load while also holding down a full time job. Then add in a family emergency and a pandemic. A C+ grade average is actually impressive in this context. But maybe next semester find ways to cut back a bit.

When I was a community manager at Stack Overflow, I often gave feedback to management that it was under-investing in community managers. Now I'm sure every department had the same feedback: if only we had more people doing X we could get more X work done. Since then, the company has reduced staff focused on community work while also increasing the scope of the team. That's just the next iteration of the bricks without straw problem. To me it's a sign that executives do not understand community management work.

It's deeply emotional labor that goes far beyond writing an announcement or "socializing" a change. It requires understanding other people and crafting messages that meet people where they are. Community managers understand members of the community are volunteers who are free to come and go as they please. They must internalize all the complicated group dynamics that sometimes result in a very bad day and chart a course that efficiently (and accurately) solves the underlying problem. In short, CMs do work that is often underappreciated and underestimated.

My current job is overseeing a community that gets roughly the same traffic as Stack Overflow en español or Stack Overflow em Português (take your pick). I've been absolutely brutal cutting things from the roadmap when it becomes clear they aren't providing value. A lot of my favorite ideas in the past 6 months were too ambitious for our team. (Doesn't mean they won't come back later when we do have capacity.) Trying lots of things (without being dogmatic about them) allowed me to discover a change that was almost trivial and had a huge impact. My timeline was:

In order to get that done, I needed support from product management, design, front-end and back-end developers, and, of course, management. We all needed to be focused on the community problem. At several points we risked over-engineering or adding unnecessary constraints to the project. In order to avoid that trap, we worked together as a team with a common goal. Because the goal revolved around community engagement, it was critical that I had input in the project from start to finish.

Nobody asked me, but I would have skipped most of the items on the Stack Overflow roadmap in favor of:

  1. Repair relationship with the moderators (not on the list, but behind some of the initiatives)
  2. Site satisfaction survey
  3. Developer survey
  4. New question experience
  5. New close experience
  6. License update
  7. Following questions
  8. Dark mode

Some of the items further down the list take less effort than the first few items. Maintaining relationships with moderators was the bulk of the community team's job when I was first hired. Needless to say repairing damage is much harder. It's exponentially harder when someone other than the community managers are making the decisions. So many of the items related to moderators bear the marks of other departments and their priorities.

The work on the surveys should be more than just coding responses and reporting results. Those tools have the potential to identify concerns from people who don't have the inclination to report them on meta (or to verify that meta reports are representative). It might be necessary to drop other items to take care of problems the surveys help identify.

For all the talk about software and technology, every one of Stack Overflow's business products depends on the goodwill of the community. The community has been taken for granted (as in assumed to be an available resource for the company to draw from) for so long it no longer understands the unique value provided by a small city of volunteers. Over the last few years the city government has invested in monorails. There's still time to fix the Main St. potholes if leadership acts now. Otherwise, I hear Shelbyville's shopping district is pretty nice.

| improve this answer | |
  • 1
    What about dark mode makes it on your top 8? – George Stocker Jul 20 at 20:39
  • 22
    You may recall, I used the phrase "bricks without straw" in a call with an exec back at the start of 2019, when we were informed our team was losing executive representation. I got... The verbal equivalent of 🙄. And of course, the CM team is now half the size it was then, which was already half the size it needed to be. I feel... Incredibly bad for my former colleagues. – Shog9 Jul 20 at 20:51
  • 2
    @GeorgeStocker: Mostly from the survey results. (I can't find what I remember, so they might not be published.) For a small number of people, dark mode is a big deal. As long as it's opt-in, there's very little downside to making it happen. Unless there are bugs or it takes away from other projects, of course. But it's also the first thing to cut if the team is running low on bandwidth. – Jon Ericson Jul 20 at 20:51
  • @Shog9: Yup. 💯 – Jon Ericson Jul 20 at 20:53
  • 1
    @GeorgeStocker to me for example dark mode was kind of breakthrough as it demonstrated that the way to increase engagement is not in making it easier for new askers to ask (low quality) questions. Until recent, company wasted lots of effort and did much damage to content by focusing exclusively on comfort for new askers - and dark mode made a brilliant proof that this was the wrong way and opened a new, more promising direction. Of course there still can be mistakes on a new way too ("thanks reaction" comes to mind) but generally it's not a dead end, as opposed to bending over to new askers – gnat Jul 21 at 8:45
  • 3
    (to avoid misunderstanding, helping new askers deserves some effort - but not on the grounds of a wrong idea that it may somehow substantially increase engagement) – gnat Jul 21 at 8:58
  • 1
    @gnat something has to be done to engage and encourage hi-rep users to post answers, and I don't think "dark mode" was or can ever be that solution. For years I've been hearing about LQQs, homework questions, and the "gimme the codes". If hi-rep users are bored and frustrated by the new content, how does the dark theme help to improve the situation? It's purely aesthetic, perhaps much sought after, but we're still talking about cosmetic improvements. – Mari-Lou A Jul 21 at 9:25
  • 2
    @Mari-LouA this is explained in my prior comment, "company wasted lots of effort and did much damage to content by focusing exclusively on comfort for new askers - and dark mode made a brilliant proof that this was the wrong way..." Dark mode did not contribute to what you want directly, but it removed a big obstacle in achieving this. Whether this will lead to changes you want I can't tell but I believe that these changes were simply impossible before, when company focused efforts on this fruitless idea of exploiting new askers to increase engagement (at the expense of content quality) – gnat Jul 21 at 9:40
  • @gnat: I'm glad you see that's your takeaway. It isn't mine, but that doesn't mean you are wrong. – Jon Ericson Jul 21 at 20:13
  • @Shog9 "And of course, the CM team is now half the size it was then, which was already half the size it needed to be." I'm always for more CMs and programmers but playing devil's advocate one could say that the network kind of runs stable currently. As long as the company doesn't start firing mods for no apparent good reason again, I would say that they are safe for the immediate future. A C+ with a reduced CM team might actually already be an achievement. There is some progress and it feels bigger to me than the one in 2017 to 19. – Trilarion Jul 22 at 19:23
  • Living cheaply is easily done, @trilarion. Living sustainably is harder. The crash last year tells one story of bankruptcy; time will tell how long the current approach will last before the debts mature. – Shog9 Jul 22 at 20:23
24

Thanks for this, Shog. I'm glad to see you're still engaged, despite no longer having any obligation, either legal or moral.

As others have mentioned in the comments, I'd grade less generously, comparing the current status quo to where it should be, rather than where it was at its nadir.

Having said that, you've put more thought and effort into this than I am willing or able to do, so I won't attempt a scorecard of my own.

I will say that I see many of these issues as symptoms of the underlying disease, which still hasn't been addressed: identifying, and clearly, honestly, and forthrightly expressing the fundamental vision of the site.

In particular, saying out loud that there is and always will be a tension between "a Q&A platform for anyone who codes" vs "a long-lasting reference for programming questions", with a clear-eyed analysis of the trade-off between attracting neophytes and retaining experts.

To date, I have yet to see the company admit to itself that no, it cannot have both.

Until that happens, diagnostics like this are just a prognosis for a patient in hospice; sure, we can continue to monitor the symptoms, but the disease will kill the patient anyway.

| improve this answer | |
  • 2
    Thanks, Dan. Regardless of the somewhat fanciful letter grades assigned, I'm fully aware of the long, hard road the company has yet to walk. They're on step 0 of 12 - but, at least they're somewhere. – Shog9 Jul 19 at 16:20
  • 12
    Yes; I agree they're on step 0 "admit you have a problem". I just don't think they've done that yet, and I've all but lost hope they ever will. You peg the problem as pride -- that's certainly one pressing problem -- but the one I peg I think is more urgent: recognizing they're trying to square a circle. Pride, correctly channelled, can be an asset; not so much for self-delusion. – Dan Bron Jul 19 at 16:25
  • 8
    That's step #1. Step #0 is "don't die" – Shog9 Jul 19 at 16:29
  • 1
    I don't see a likely, acceptabile to the corp., source of Good Orderly Direction either. – User777088 Jul 19 at 16:39
  • 4
    @DanBron It's common in education (esp in grade school rather than university) to at least consider grading effort and improvement, rather than final product. For a struggling student, failing them/giving low grades doesn't send a useful message, they just get dejected. Similarly, if the best students get As with minimal effort, you aren't pushing them to be their best. (note: good educators use grades as tools, not just competition). I think it's fine to give generous grades to a troubled company; that doesn't mean you don't expect better work in the future to get the same grade. – Bryan Krause Jul 20 at 2:22
  • 3
    @BryanKrause That games only works if (a) you can communicate the situation to the student and (b) the student cares about the grades. The evidence for the c-suite paying attention is limited and for their caring much is even thinner on the ground. – dmckee --- ex-moderator kitten Jul 20 at 17:33
  • 1
    @dmckee---ex-moderatorkitten In that situation of limited attention resources and if you are unsure if the student cares, do you think giving a slate of all Fs be more effective, just as effective, or less effective in helping the student grow? – Bryan Krause Jul 20 at 17:35
  • 3
    @BryanKrause In all honesty that depends on how the student relates to the results and to you. Sometime a shock can be more effective than the nice guy approach; sometimes it would be disasterous. But a corporation isn't a child or a teen. The people who you want to influence are grown ups. They are alleged to be hard-eyed realists. Should we soft-pedal the evaluation for the benefit of people making six figures and holding an equity stake? – dmckee --- ex-moderator kitten Jul 20 at 17:47
  • 1
    @dmckee---ex-moderatorkitten It's not necessarily for their benefit in this context, ultimately, but for ours. If the community just keeps saying "you are doing it wrong" without pointing out where some steps have gone in the right direction, then the community is failing to give useful guidance. Instead, they're just venting frustrations, which only has short-term benefit. And the psychology doesn't apply just to children/adolescents; you'll find the same approach in interpersonal counselling or employee management. – Bryan Krause Jul 20 at 17:58
  • 3
    Efficacy of actual pedagogical techniques not withstanding, I feel the need to remind y'all that this was just a game to keep me entertained while reviewing the roadmap - I'm not the company's teacher, and I am fairly certain nobody in charge at SO gives a hoot about anything I write, much less whether I give them an "A" or an "F". – Shog9 Jul 20 at 18:58
  • @Shog9 Not the people in charge necessarily (unfortunately), but I suspect a few of the employees do. It may be questionable what things they have control over or what voice they have, of course. – Bryan Krause Jul 22 at 18:52
21

I am not sure if this is Jack's fault, but IMO grades are a bit too high.

The only things that deserve A are Follow Questions and CC License Update update (it came late, but it is the only thing that was adequately resolved)

For the rest, I don't think that anything deserves more than C+ (maybe some things could get a B)

But, overall progress in individual items is still being in the shadow of one enormous issue: accountability

I am sorry to say, but you cannot solve issues by introducing new person that took no part in the events and say "we are starting with clean slate". It does not work like that.

The only way primary issue could have been resolved was to reinstate Monica. Anything else leaves huge hole in the site's foundation and cannot be resolved by adding new coats of paint.

Teresa had the unique opportunity to make (some) things right and she didn't take that.

Accountability never happened and it seems like it never will. There is nothing in CoC, Moderator Agreements, and all other processes that can fill that hole. They all require a certain amount of trust among all parties, and without resolving primary issue there is no trust to begin with and to build upon.

| improve this answer | |
  • 7
    There's no retroactive resolution for any of this. The license stuff is nice for folks still around, but small comfort to the folks who deleted their accounts over it during the two years it wasn't resolved. I don't see the company agreeing to refund all the folks who donated to Monica's legal fund even if a reinstatement did happen, much less finding a way to fund all the projects that were dropped, bring back all the people who left. Starting fresh never really happens; sometimes ya just gotta... Start again. – Shog9 Jul 20 at 12:36
  • 4
    Of course, not everything broken can be made whole again, but that does not mean that nothing should. If there was one thing that could have been done, that was reinstating Monica and that act of good will cannot be replaced with anything else. – Resistance Is Futile Jul 20 at 12:55
  • 5
    No disagreement there. But I personally lost all hope on that front a long time ago. – Shog9 Jul 20 at 12:56
  • 2
    Me, too... but this is not so much about hope that will happen, but pointing the reason why any future step will be looked from the worst POV and dissected to oblivion. – Resistance Is Futile Jul 20 at 12:58
  • 5
    Your and other CM's departure presents more critical failure and will have much deeper impact in the long run from the POV of what company lost and will be capable of achieving from the inside, but from perspective of building the trust between community and the company, Monica is more essential. Again, circumstances around your departure don't help that situation either. – Resistance Is Futile Jul 20 at 13:02
  • 7
    There are folks who could do what I did better than I ever could. Of course... If the company couldn't afford me, then I don't see them getting or keeping replacements either. Right now they're depending on training and overworking a tiny team with, somehow, more managers than workers... I don't see that being particularly sustainable either. The loss of a person is sad; the continued existence of the factors that led to their loss is tragic. – Shog9 Jul 20 at 13:08
  • 7
    I think you are selling yourself short, really, really short ;) and FWIW, I am glad that you are still around. – Resistance Is Futile Jul 20 at 13:18
  • Thanks, that means a lot – Shog9 Jul 20 at 13:20
  • 2
    Reinstating Monica would be the same as admitting they were wrong. The closest we'll ever get is a series of non-apology apologies ("we're sorry this happened", "we're sorry people got offended by our actions"), etc. – Richard Jul 20 at 15:04
  • 3
    This is practically beside the point but you can indeed follow answers (it was even in the announcement post, I guess they expanded scope from the original roadmap item) :) (cc @Shog9) – Em C Jul 20 at 16:38
  • 6
    Good eye, @EmC - I have a userscript installed that screws with those links (and, also, doesn't work anymore); my bad. I've corrected the post! – Shog9 Jul 20 at 16:44
18

Noticeably absent from your report is the CommonMark migration. Now... I know you mentioned in a comment...

I'm very happy with the switch to CommonMark, but... Weirdly enough, that wasn't on the roadmap!

... and that's true! Though I would argue that this post was tagged for a good while before, during, and for a bit after the migration occured. In fact, it was featured for the entire month of June (tagged featured in this revision, untagged in this revision.)

The question of why it wasn't in the roadmap is a fair one, but it's not like we didn't see this one coming. Additionally, it occurred over the course of last month, so I feel it's fair to bring up in a post such as this.

Let's give the migration a review!

The Intro

Ham Vocke goes to great lengths in the migration's announcement to elaborate on what was going to happen, when it was going to happen, why it's going to happen, and provides a thorough explanation on what the migration entailed for the network as a whole. Ham answered most every concern that emerged in the form of an answer to the announcement post, and tried as hard as possible to ensure everyone was informed before migration even began.

This, to me, is excellent. Realistic expectations were created and adequately explained the community... How's the follow-through?

The Migration Itself

Before migration, Ham created a consistently-updated schedule underneath the announcement, which was updated every time another site migrated.

Meta Stack Exchange (as is usual) was the first site to move over. Very early on, concerns, bug reports, and other issues were brought to the attention in the form of answers to the announcement post, or other bug reports altogether. Some of those issues involved the CC licensing. In a large batch of fixes, a backfill was run, cleansing away the licensing changes on relevant revisions as well as fixing a large portion of the CC licensing issues... all in one fell swoop.

Other problems propped up, but Ham Vocke was on top of it. Ham responded to almost every single answer under the announcement. Whether they be general inquiries or bug reports, Ham was almost certain to reply.

Credit where credit is due... Ham was everywhere.

When the time came for Stack Overflow to be migrated, Ham created a Meta SO post. It seemed largely unnecessary, as most of the issues with the migration were discovered long before Stack Overflow was slated for migration... but it was still made! The two bugs that were posted there were , and served as a learning experience for future readers as to the nuances of CommonMark.

The Finale

The CommonMark migration finishes, and the announcement post is left to be featured network-wide for another week post-conclusion to ensure visibility.

Final Review

I'll use your grading scheme:

  1. Completion: Full marks here. Not only was the task completed, but we received copious updates as to its progress throughout the entirety of its implementation.
  2. Quality: Bug reports were addressed, support questions were answered, and you and I both wrote these posts using CommonMark's markdown. I'd say this task was well done.
  3. Reception: Tell me if I'm delusional... but I'm pretty sure the migration was well-received. At +571 -10 on the announcement post on Meta SE and 79 average on Meta SO's announcement, I'd say that's a pretty good score when all is said and done.

Should this have been on the roadmap? Absolutely! I think so, too; but even without it being mentioned in the roadmap, I think the CommonMark migration earns a rightful...

A

| improve this answer | |
  • 9
    No disagreements with any of this - a resounding success! Hopefully we'll see more stuff like this on the 2H roadmaps. – Shog9 Jul 21 at 17:36
  • Fingers crossed! – Spevacus Jul 21 at 17:51
  • 8
    Ham was everywhere indeed. In particular, Ham was on tex.se, where the community was very concerned and did not trust the company to clean up the mess that might be caused by the migration, mostly since the company made a massive mess the community had to clean up, in the past. – Discrete lizard Jul 21 at 18:00
  • 14
    Faced with such animosity, Ham did not fear to actually engage with the community properly, including: 1. Listening and acknowledging how the community feels, without blame or making light of their worries. 2. Acknowledging the concerns and give as much transparency as reasonable can be done 3. Being realistic that mistakes may be made, but assuring they can be fixed. This is how it should done. Extra credit for Ham! – Discrete lizard Jul 21 at 18:00
  • 2
    I wonder what more is needed to get an A+ score. – Bart van Ingen Schenau Jul 22 at 13:02
  • 1
    Agreed, Ham Vocke's steady communication around the CommonMark migration was great. :) – V2Blast Jul 28 at 8:37
18

I don't think it's particularly fair to really just look at the last 6 months or a year. SE's been at best a straight E student the past half decade. I hope we keep up the momentum we've gotten so far, but getting anywhere close to a D or C overall is going to take a lot of mugging

Dysfunctionality isn't something that happens overnight - and the signs of the things that came to a head in late 2019 were obvious earlier. SE's lapsed into what was could be described as "benign neglect".

I think a promise was made that things like careers would result in things getting better, and that SO was the "shining city on the hill". What we seemed to end up with was the parts of the business that were supposed to be the engines of growth ended up sapping resources from the network - we lost folks on the front lines of our sites cause of careers wasn't quite what it was hoped to be. We need them to succeed, but not at the cost of our Q&A sites.

I suspect the key factors of success are somewhat more than what's there in the list

  • The "commercial" side of the company needs to be at least self sustaining. We shouldn't be in another situation where a retrenchment event is triggered by one of those business units. In addition, there's nothing unique to Careers, Enterprise or Teams without the userbase here. We certainly shouldn't be losing Community Team or devs or SREs for growth (or lack thereof) of those business units.

  • There are a lot of active diaspora communities from previous blow ups - getting them back is a start. They need to feel this is the the SE they came to in the first place.

  • Having a community team that has the resources and bandwidth to engage the community, they're good folks, but stretched so thin you could read a newspaper through them. SE grew up with a great, well resourced community team. Essentially they need to be empowered and trusted to do the right thing. No pressure from above, and let them sort out things before they get too bad.

    This confuses and saddens me. Jeff got it (He kinda started the CM team, and acted in some capacity with them). Joel seemed to get it. Yet here we are with not enough CMs, lots of messes that could have been avoided, and a company that needs poking with sharp sticks to remember things that once were second nature.

  • Regaining the trust of the community. There's one big thing that might help, but the damage is deeper than the big issues. This will take a willingness to invest time, resources, and a willingness to accept that some things went wrong, and to fix them. We'll help you know what they are so you don't know what it is the next time, but old wounds still crop up a lot.

| improve this answer | |
  • "SE's been at best a straight E student the past half decade." This implies the problems are from much earlier than suggest in the question and the recent dumpster fires, from back in the day we did have a decent CM team. – Mast Jul 20 at 14:38
  • 7
    Well yes. 2017 is when the cracks showed - but those problems started earlier. The focus on SO and careers, that there was little development on Q&A (which ended up being troublesome when development kicked off again), the general attitude of the powers that be going from "one of us" to "we tell you what to do" . A functional CM team probably kept the issues from being worse but problems to me feel much more fundamental. – Journeyman Geek Jul 20 at 14:42
  • Fair enough. It just seemed like your answer insinuates a bigger/better/stronger CM team would have made a decent difference and that Joel seemed to get it. Yet from the past half decade, much of that was Joel-time. Looks like we didn't fare much better under that either if I read your comment right. – Mast Jul 20 at 14:43
  • 7
    @Mast Joel basically both got the CM team to its largest (and instituted team CHAOS) and did lots of good things. He also kinda ... didn't seem to do much near the end, and let things get as bad as they were. I see Jeff's time in charge with rose tinted glasses. I see Joel's time as having a promising start, and being the root of many of the issues we have now. – Journeyman Geek Jul 20 at 14:46
  • 6
    So - a strong community team's part of the solution. But we also need the other parts of the company to actually succeed and be expected to succeed, without passing on the damage of not doing so to other parts of the company. We lost CMs not because they failed, but other parts of the company had trouble. There were issues with culture high up, which resulted in friction with a lot of smaller sites since we were underappreciated. Some realised this but I don't really think they could do much. – Journeyman Geek Jul 20 at 14:48
  • 2
    @Mast 2015 - 2018 – gnat Jul 20 at 16:36
14

Like you, we are strong believers in the power and importance of actions. As you and others have pointed out, we spent a lot of time in this year talking about what we intended to do. We also spent a lot of time trying to deliver. It may not be apparent for those watching from the outside, but each one of these initiatives (regardless of the grade that you and your ad hoc committee assigned) took many many hours of work, and involved more people than you might have expected. I am proud of all the work that my colleagues and I have put in this year to get to where we are and would like to recognize their efforts.

We have had successes, we have had failures (and in between as well). I appreciate your attempt to call out everything as objectively as you could. I know that for everyone who is invested in the network and feels that they are a part of the Community, there has been pain, and there has been a loss of trust — and it is up to each individual as to whether they feel that that loss is irrevocable, or if a recovery is possible. And any possible recovery and re-bridging must be based on open communication, on acknowledging both successes and failures, and most importantly (in my opinion), on both working to ensure that the recipes for success are emulated, while the lessons of failures are learned well and not repeated. I am happy to report that we have some very firm processes in place now through which we review things that didn't go well, and institute concrete steps for learning from these things in the future.

So thank you for continuing to care, and for being so vocal in your attempts to honestly talk about how things went — both the good and the bad — and for giving your thoughts on where things can and should change. Reading the different responses here (and elsewhere) makes it evident that while different folks have different opinions about how things should proceed, all of you continue to be invested in what is going on here, and interested in seeing the relationship between the Community and the Company continue to heal.


I am not going to comment on any of the individual grades that you gave. Your report card; Your decision. Though I do find it interesting how for some categories it is literally impossible to score an A due to carry over sentiment from previous terms (all report cards that I received in my educational career for a term were always based solely on the efforts made during that term).

That said, I will comment here on some of the specific items where I am able to provide some context that you were not able to via previously published materials (just writing about what I know - please don’t draw conclusions based on what I am not commenting on):

  • Site Satisfaction Surveys: these have been ongoing for over a year on Stack Overflow, with over 1000 respondents every month (split between logged-in and anonymous). Yes, we haven’t published many of the results, and hopefully that will change. I can say that collation and examination of these is done on a continuous basis, and the results of this have and continue to factor significantly into our roadmap planning process.
  • The Loop: tag has been added to the last one.
  • Internal Community Monthly Newsletter: is published every month, and has gotten good internal feedback, is doing a good job at keeping folks informed of things going on with the Community who otherwise would not hear about them.
  • Community-a-thon: this ended last week, and I am planning on posting about it once all of the post-event surveys are collected, and we have had time to sift through the feedback (the intention to follow up was noted on the canonical post). I can say right now that I consider the event to be a success: almost 80% of the company participated, with good representation across all departments, length of tenure, and seniority within the company. Many staff members posted for the first time ever, and we have dozens of pieces of feedback from users covering all areas of their participation. Look for more details in a follow-up post in the future.
  • Moderator Reinstatement update: I take issue with the assertion that "there's no denying that the reception was bad". The reception on MSE was not good. However, the reception from the Mod Council and the Moderators as a whole was on a whole positive, and very different in tone and numbers from the reception on MSE. As the mods represent the actual user group to whom the policy applies, this doesn't feel to me as unequivocally bad as your summary implies.
  • CC License update: I am unaware of any outstanding kinks. There was a follow-up update sometime ago that addressed all major outstanding issues that I am aware of. The only remaining related item that I am aware of is to make a timeline page with license notations for tag wiki and tag wiki excerpts, and this is already in the planning stages.
  • Review Queue Phase 2 updates: first release and posts on MSE coming this week

So - thanks for the C+ and for taking the time to organize your thoughts, for continuing to keep us accountable.

| improve this answer | |
  • 10
    I think the differences in how the reception of the reinstatement update was perceived is a good reminder that most of the community only sees part of the story if an effort isn't made to share what goes on internal to the company and in moderator-only spaces. I appreciate your answer, but I would like to toss a yellow flag out for As the mods represent the actual user group to whom the policy applies.. Almost all of the community are potential moderators and their concerns about official policies shouldn't be dismissed out-of-hand as less relevant. – ColleenV Jul 21 at 15:02
  • 9
    And also, the SE community is full of talented people experienced in a diverse range of disciplines with global perspectives. It would be a shame to make them feel like their feedback on moderation policies isn't welcomed because they aren't currently moderators. Process for removing and reinstating moderators is not such a unique problem that no-one could possibly have any valuable insights into it other than current SE moderators and staff. – ColleenV Jul 21 at 15:07
  • 5
    There are some schools that base their grades on the relative improvement and achievement of the partially self-set goals of the student, although this is more of an exception rather than the norm in any place I'm familiar with. In any case, this isn't school, and anyone who has been on SE longer than today will judge the company based on their history with it. It can't be otherwise: none of us can see the full internals of the company, so we must take our mutual history into account to predict what will happen in the future. – Discrete lizard Jul 21 at 15:08
  • @Discretelizard I hear. It is just that the post is titled "2020 Q1/2 report card" - and all of the topics are for Q1 and Q2 commitments. So one might think that the grading is referring only to work done during that time period. – Yaakov Ellis Jul 21 at 16:39
  • @ColleenV not intending to dismiss non-mods out of hand. Just to point out that "the reception was bad" might not apply to all reviewers. As far as the feedback from non-mods on mod-related policies, I think that you will find (if you look at the relative posts) that we welcome it, and respond to it quite often. – Yaakov Ellis Jul 21 at 16:41
  • 2
    I understand that, mostly because I have been a moderator. I was trying to alert you to how things look different to people on the other side of the curtain. I think it's possible to say "I can see how you might think that, but here's more information that might cause you to view it differently." without discounting the negative feedback as less relevant. – ColleenV Jul 21 at 16:43
  • 1
    @YaakovEllis Fair enough, but if you'll allow me to play devil's advocate for a moment, while the current grades in principle are about the material covered in Q1/2, mastering the prerequisites of the lessons taught last year will be nessecary to achieve a good grade. A student's grade is about their current ability, yes, but this is not only formed by what they did during the last semester, because while the courses are discrete chunks, a student learns (well. . , should learn) continuously. As with the student, so with the company. Keep at it and you will likely get a better grade next term! – Discrete lizard Jul 21 at 17:07
  • 12
    Factual error: the reinstatement process does not apply to current moderators. By definition, it applies only to former moderators. How many of them did you talk to? I've spoken with... Several. – Shog9 Jul 21 at 18:11
  • 1
    @Shog9 I stand corrected. Of course, it only applies to former moderators, all of whom used to be in the pool of moderators. I have also spoken to former moderators. But this is getting off topic. This is your only reaction to what I wrote? – Yaakov Ellis Jul 21 at 18:15
  • 17
    Every announcement tells a story. The story of the roadmaps was "we're working to make things better". It was the right story to tell. In the long run that story only works if people feel the company is making progress. I see progress toward repairing relationships with moderators, so the story rings true. But I also know the only reason there was progress at all was because there was hardly anywhere to go but up. I'd rate things a lot lower if not for considering progress from the previous term. We (and I include myself because it was my job then) deserve academic probation for last year. – Jon Ericson Jul 21 at 20:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .