7

May’s “The Loop” blog post was just posted, you can read it here. A popular response last month to our post about how we receive, prioritize and implement feedback was a question around how this framework gets utilized in the practical sense. We thought that Dark Mode was a great feature to dig down into as it was so widely requested and anticipated.

We hope you enjoy this month’s post, please leave any questions or comments you have in this thread and we will respond ASAP.

3
  • 7
    Since it's about dark-mode, shouldn't this be posted on Meta Stack Overflow? </jk>
    – Glorfindel Mod
    Commented May 26, 2020 at 14:50
  • 5
    We thought about it, @Glorfindel. In the end since it's about the feedback frameworks we use across the board, we thought it might still be interesting to the folks here on MSE.
    – Sara Chipps StaffMod
    Commented May 26, 2020 at 14:52
  • Is there a plan to have this on other sites as well? I know there are differences, but some time ago sites switched to common design, that could make it easier perhaps... And just noticed that post of @RobertColumbia below and comments there seem to address this.
    – Sil
    Commented May 29, 2020 at 9:50

5 Answers 5

34

Since we’ve launched Dark Mode, our data team has been heads down on analyzing the effect to the network. Not only has it led to the most signups Stack Overflow has experienced in a month, we’ve also seen a rise in engagement around the network.

It seems odd that the analysis suggests Dark Mode led to a huge increase in sign-ups. I... think there's an issue of conflating correlation and causation here. The huge increase in sign-ups as well as engagement happened at the same time that a majority of the world was directed to work from home -- where now IT policies aren't as strict and more people can A) use StackExchange while working and B) have more fluid integration of free time and work time, possibly leading them to look up something for work and take a few minutes to answer questions or otherwise participate.

I doubt Dark Mode is leading to a huge increase in signups and engagement. It's much more likely that pandemic-induced change in work patterns is the driving factor. But either way, the data does not lead itself to the conclusion that Dark Mode had any impact because it isn't controlling for other, conflated changes.

I'd suggest not buying into that conclusion too deeply without a lot more work to separate other possible causes.

14
  • 8
    I also was surprised by this correlation at first but eventually figured what's the matter. Having dark mode requires account (because it's in account settings). And, although maybe only small part of site visitors wants it strong enough to make an account, this is part of really big BIG audience, part of folks making those millions daily visits at Stack Overflow. And this small part of a very big audience turned large enough to make a solid impact
    – gnat
    Commented May 26, 2020 at 17:13
  • @gnat Alternatively, only a few states issued stay-at-home orders prior to the week before Dark Mode release, and in the run up to the release, the rest of the states that would issue orders did issue them. Similar lockdowns were also rolling across Europe. While your explanation is a plausible justification for why it could be true that Dark Mode led to improved metrics, the data (at least as presented) doesn't support or disprove it. I'd tend towards the simplest explanation as most likely (although again, unproven) -- more people at home == more engagement. Timing is probably coincidental.
    – tpg2114
    Commented May 26, 2020 at 17:16
  • 1
    I guess to sum it up... the data can't disprove the null hypothesis and so the conclusion isn't supported.
    – tpg2114
    Commented May 26, 2020 at 17:19
  • 1
    I think my theory can be verified quite easily, by checking how many of the new accounts have tried to change to dark mode setting. Depending on what part of them did that we can conclude how plausible it is
    – gnat
    Commented May 26, 2020 at 17:21
  • @gnat I would argue that's just correlation and not causation again. The first thing I do when I create an account somewhere is poke around the settings, and I'm sure I'd go "Oo, a dark mode" and turn it on. But that doesn't prove that's what motivated me to create the account. I think there would have to be a properly-constructed survey to identify the motivations, which would first require formulating the possible alternative hypotheses and then writing non-leading questions to suss out what the motivating factors were in creating the account.
    – tpg2114
    Commented May 26, 2020 at 17:23
  • 10
    The change literally is a gigantic spike the exact day we released Dark Mode. If it were based on increased usership due to covid, it'd have been gradual ramp up over the weeks prior and after... There wasn't a huge global event that happened exactly on March 30th.
    – Catija Staff
    Commented May 26, 2020 at 17:24
  • 3
    @Catija How much of the spike is users on the network without SO creating an account to see Dark Mode and then going back to the rest of the network? It's certainly possible that Dark Mode did have this effect -- but just plotting the spikes doesn't prove it. There's a lot of conflating variables and looking at signups vs time doesn't control for all of the other conflating variables. There's a lot of statistical tools to help determine this, and maybe that's being done -- but the data shown in the blog doesn't definitively prove the conclusion.
    – tpg2114
    Commented May 26, 2020 at 17:30
  • 1
    ... But... that explanation doesn't mean that Dark Mode wasn't the driver... you essentially just said that Dark Mode was a big driver. We're not saying (or at least I don't think we are) that what you're guessing is wrong but it's still related to Dark Mode. I think it's pretty easy to check whether an account was brand new that day or whether it was just a new SO profile on an existing account... but I'm going to guess, regardless, that the spike occurs in both groups. And if someone just joined SO to see the new shiny, they wouldn't continue to use SO after... and people are.
    – Catija Staff
    Commented May 26, 2020 at 17:34
  • 2
    Other sites have done the same thing - made dark mode require an account - and they see increased engagement when logged in because people sign up and stay logged in to take advantage of their dark caves' not being infiltrated by the scary light ;) Anyway, I don't think the blog is about proving that Dark Mode increases engagement. It's about showing how we've released a feature and how that release went through the phases Sara outlined. :D
    – Catija Staff
    Commented May 26, 2020 at 17:35
  • 2
    @Catija Totally agree, it doesn't mean it wasn't, but I'm also arguing that the spike in signups doesn't mean it was either. Likewise, spike in engagement elsewhere doesn't indicate it is Dark-Mode-induced vs other variables. I guess my point in all this is the data shown doesn't prove the conclusion -- it's suggestive, sure, but not proof. Proof would require alternative hypotheses, additional data, and demonstration that all of that together means Dark Mode is more likely than the other hypotheses.
    – tpg2114
    Commented May 26, 2020 at 17:38
  • 2
    What I'm saying is, without the p-value and more data, the two figures and description on the blog don't prove the conclusion (p-value is a stand-in for whatever method is needed to prove statistical significance and to disprove the null hypothesis). That's my feedback -- the statement made is way stronger than the figures support. Maybe all the hard work has been done already and just wasn't presented.
    – tpg2114
    Commented May 26, 2020 at 17:39
  • 2
    Ah. Yeah. I think there's likely more data. Not sure if the data team is planning a deeper dive. I think Jason posted some other info in comments on an answer - meta.stackexchange.com/a/347560/284336
    – Catija Staff
    Commented May 26, 2020 at 17:45
  • 4
    The graphs seem to imply that a lot of lurkers have signed up and remain signed in. That doesn't necessarily mean there will be broader increases in engagement. Commented May 27, 2020 at 3:36
  • 2
    @curiousdannii these graphs also imply that if company wants to increase engagement metrics (to impress investors for example), they may have reasonable luck focusing on attracting millions of lurkers who use Stack Overflow for its primary purpose (which is finding already posted answers to already asked questions). Ideally, focusing efforts on this would be combined with shrinking (long proven useless) efforts on improving said metrics by attempts to welcome comparatively miniscule amounts of new question askers (who often tend to just dump their homework into question box and run away)
    – gnat
    Commented May 28, 2020 at 12:38
15

You mentioned paying off a lot of technical debt to implement Dark Mode on Stack Overflow. How much of this effort would be useful in expanding Dark Mode to the entire network? For example, would you need to re-implement it mostly from scratch on each individual site, or is there now a Dark Mode Module that can be installed, tested, and tweaked pretty much anywhere you want to with a much smaller amount of effort than was necessary to implement it the first time around for Stack Overflow?

2
  • 11
    From talking to Aaron... it's kinda complicated. We can't just turn a lot of sites dark because their artwork was designed with light mode in mind (excepting for Science Fiction & Fantasy, for the most part). So plopping dark mode on, say... Worldbuilding or English Language Learners or Seasoned Advice... would be pretty impossible without redoing their artwork... again. This is one of those... rare cases where beta sites have a leg up... :)
    – Catija Staff
    Commented May 26, 2020 at 14:58
  • 7
    Before concerning themselves with useless dark modery there's more important things to do, including general non-dark design work. Enough with dark mode already! :'( Commented May 26, 2020 at 15:27
14

Cross-post from “The Loop May 2020” contains inaccessible image with no transcription, which was closed as a duplicate of this question.


In the latest blog post, The Loop, May 2020: Dark Mode, there's an image directly after the first line. It contains a chart and looks like it contains useful information, which is nice.

However, if you can't see images (or are using a screen reader or something), then... you can't access any of the information on the chart. This is what it looks like if I disable images:

Screenshot of the blog post, showing the first line of text, a big blank space, and then a continuation of the text.

This is a regression of sorts; in the Q2 Community Roadmap blog post, there was a very nice chart included that was not in image format (which has the additional benefit of being able to copy-paste, and of being dark when dark mode is enabled):

Screenshot showing text chart in the Q2 Roadmap blog post

Could you please either at least add alt text for the image in the May 2020 Loop blog post, or, preferably, abandon the chart-in-images thing and just use the text charts?

7
  • 1
    Such tables don't scale well on mobile, guess that's the reason they use images instead. Agree about adding alt text though, that's a basic requirement. Commented May 27, 2020 at 7:13
  • 1
    Alt tags have been added. We'll try to be better with this moving forward.
    – Yaakov Ellis StaffMod
    Commented May 27, 2020 at 7:16
  • @YaakovEllis - thanks!
    – Mithical
    Commented May 27, 2020 at 7:32
  • @Yaakov that's awesome! Was the text ready and just not put in proper place, or did you add the text now yourself? (Not that it really matters, just me being... me. ;)) Commented May 27, 2020 at 7:38
  • 3
    @ShadowKeepsSocialDistance Yes
    – Yaakov Ellis StaffMod
    Commented May 27, 2020 at 7:40
  • 1
    The table has also been converted to html
    – Yaakov Ellis StaffMod
    Commented May 27, 2020 at 9:45
  • @YaakovEllis - Woo! Thanks, that's great
    – Mithical
    Commented May 27, 2020 at 9:46
9

I feel a bit torn here.

On the one hand, when some app/OS supports "dark mode", I typically give it a try, and very often (not always) stick with it. So, yes, having that choice is a very useful thing. And yes, I understand that it was a lot of effort, and that maybe other people were really waiting for it. So I get why you dedicate so much "blog space" talking about it.

But then: I think I never abandoned a tool because it didn't support a "dark mode". I pick my tools, and my online communities based on their core features, and the value they give to me.

Example: I recently stopped using quora, and sent 4000+ answers down the drain. Not because quora doesn't support dark mode, but because I got fed up with many subtle technical issues, and most importantly: because moderation there is absolutely intransparent and inconsistent. And because there is basically zero open communication between "the company" and "the community".

Meaning: I see different priorities, especially fors stackoverflow, like:

  • Improving search
  • Review queues, and the overall workflow around the queues
  • And sure: improving search (to really reduce the number of duplicate entries coming in)

Thus personally, I would have wished for these things being addressed before "dark mode".

On the other hand: neat UIs attract users, and users ideally translate to revenue, which pays your salaries and the infrastructure that runs our content. So I accept this (imho: over)focusing on "dark mode" as real world necessity.

Beyond that: I very much appreciate that blog post. I like how you use that specific feature to show us how you collected different sorts of feedback.

That is the thing that really matters to me here: that you are transparent about that process!

Keep doing that, and we will be with you.

And yes, I agree with other users: I very much doubt that your new "dark mode" send so many new users to stackoverflow. I find it more likely that "corona" driving millions of people into "isolation" has something to do with that spike.

2
  • 1
    Thanks for your feedback. For what it's worth, most of the opportunity cost spent here was taken from other design-related initiatives. The main one in line being to make progress with responsive design. And a good deal of the work put in here for dark mode (converting things from old design to our new Stacks framework) is stuff that moves us forward in responsive design (which needs that work done anyway). This did not postpone review queues (and sorry to say, that while search is definitely in need of some love, it is not on the immediate roadmap of things that we will be working on).
    – Yaakov Ellis StaffMod
    Commented May 28, 2020 at 12:44
  • 2
    Just saying: search not providing good results is (imho) one of the main causes for too many duplicated questions. Any bad question that gets asked and that needs reviewers to process them costs. Any bad question that doesn't get asked because search worked and prevented the question from being written ... does NOT cost. Thus I think your priorities are misguided there. Dont invest all energy into fire-fighting equipment while ignoring the fire-preventing stuff.
    – GhostCat
    Commented May 28, 2020 at 13:56
4

In April 2019, a user opened a GitHub issue for a dark theme on our repository for Stacks, our design system for Stack Overflow. It wasn’t the first time we’d heard the request. Having a design system like Stacks made this much more of an attainable feature. While there was still a lot of debt to be paid down, a large portion of Stack Overflow is implemented using Stacks. The issue got a lot of traction quickly and became the highest voted issue to date (well, the most thumbs up emoji) all of this brought this option front of mind for lots of people.

Okay, so far so good.

Dark Mode is the #1 requested feature in the history of our Meta Stack Overflow site. It’s a thread that has gotten revisited quite a few times over the years. The topic was first brought up nearly six years ago and is one of our top voted questions on Meta Stack Overflow.

Uh, okay now I sort of have a problem with this.

The problem I have is that this reinforces the idea that you don't listen to us on meta. On meta, the issue appears to be ignored for years. However, the minute the issue is raised on Github then you decide to take action.

Now, to be fair, the original meta post was tagged as status-declined in 2014--shortly after it was asked. It's not as if it was sitting out there totally ignored for years. You guys made clear your intentions to not add it in at the time.

Personally, I believe your intentions here were good. However, it looks bad. The issue of you guys not listening to meta is still a sore spot among users. This just feels like another example of you ignoring us on meta yet again and just doing your own thing.

There's nothing you can do about this particular situation now. However, I'd appreciate it if you were more careful not to do this in the future. And, to your credit, it is clear you have tried to change your old ways. I see Yakov Ellis and other CM team members addressing issues as they come up and tagging them appropriately. So I'm personally hopeful that you have already made this adjustment and changed your ways. In that case, I recon you could sum this up as keep up the good work (responding to feature requests and bug reports) and be careful not to appear as subverting meta again.

8
  • This doesn't seem like an example of not listening to meta users. The original request was marked declined, which was an official response, and at the same time was probably removed from SE's internal consideration. If the request had been completely unhandled, that would be a case, but in this case there actually was an official response. Commented May 28, 2020 at 5:12
  • 1
    So telling us about the lengthy history that "dark mode" has on metas ... resembles "ignoring metas"? I think it is the exact opposite: they didn't shy away from mentioning how long this was requested, thus admitting how long they avoided doing something about it.
    – GhostCat
    Commented May 28, 2020 at 6:17
  • @GhostCat No, not taking action until it was brought up in a system separate from meta resembles ignoring meta.
    – Chipster
    Commented May 28, 2020 at 6:24
  • @SonictheStay-HomeHedgehog Well, I mentioned that it was declined at the time. I agree that SE probably didn't mean any harm here. I'm mostly just taking issue with not declining a similar request on Github. As the post mentions, it's not like they were unaware of it on meta. It's just that they didn't act on it. It took using system that is not meta for them to act upon it.
    – Chipster
    Commented May 28, 2020 at 6:28
  • 1
    @GhostCat I think I see your point, though. They are doing something about it now.
    – Chipster
    Commented May 28, 2020 at 6:30
  • Attitudes can change after five years. The original request was probably never noticed after that time because it had already been marked declined and had been removed from consideration internally a long while back. There's no evidence to suggest that if a reconsideration request had been filed on meta, they wouldn't have also responded positively to it. Commented May 28, 2020 at 19:36
  • That makes sense and is fair feedback. I think what happened here is it took a long time to be able to have the time and resources to attack this, which happened to align with the timing of the Github request. I don't believe that request is what pushed us to get started.
    – Sara Chipps StaffMod
    Commented May 29, 2020 at 14:26
  • 1
    @SaraChipps Sure. Personally I don't think you meant any harm by it. If anything, you guys probably thought you were doing a good thing by finally giving a us a feature already discussed on meta. It's just I'd like you guys to be more aware of that in the future. And let me say again that I think you guys are being more aware of that now, so thank you.
    – Chipster
    Commented May 30, 2020 at 20:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .