34

Two weeks ago on May 30, a change in the vote button style went live on the network. Instead of large triangles, there are now smaller triangles with larger circles around them and changed color highlights (I think). The community's reaction to the change was highly critical, with approximately 85% of voters (and >700 votes in total) being against it. The change was mostly justified by the company by a "net 28% increase in overall votes" that they allegedly observed when testing the feature.

Now that the change has been live for some time, it might be possible to get a more relevant estimation of any effect of the style change on voting patterns, because it affects the whole network now, and the novelty effect will wear off at some point.

Is the new button style really increasing voting activity significantly?

I can hardly imagine 28% change just by such a simple change, but I personally have to admit that on mobile I like the circles and they seem less bad to me than I thought initially (or I just got used to them). I would expect a smaller change in the single digit percent range if at all.

However, one needs to be careful in the analysis because many other things also changed during the last two weeks, so maybe a comparison of the two weeks before and two weeks after the change, normalized by some other quantity (like number of new questions and answers as proxy for activity), might be useful.

3
  • 10
    I would say it hasn't been that long to really know. Also the strike might skew the data one way or the other. Personally almost stopped opening StackOverflow completely while being quite active on MSE (including voting) during the strike, so my behavior has definitely changed Jun 14, 2023 at 6:56
  • 1
    @MissSkooter With thousands of votes everyday on SO alone I expect the statistics to be sufficient and I also think the effect will not increase over time (rather regress a bit when the novelty effect wears off). MSE really was rather untypical in the last two weeks, but the main sites for every stackexchange did not see any extraordinary influence as far as I know. Jun 14, 2023 at 7:06
  • 3
    That's fair, I have no evidence to back up the claim that the strike has had any influence on main sites (at least in terms of voting/answering). Intuitively I would imagine a drop in contributions, and new users can't vote anyway. But I do see the validity of your question. Would be cool if they post a followup study. Jun 14, 2023 at 7:15

1 Answer 1

51

The simple answer is no; voting patterns have not increased since this change was rolled out (at least on Stack Overflow). If anything, voting has decreased since this change was rolled out.

The "site analytics" privilege allows anyone with 25k+ reputation to investigate this for themselves. Having someone with 25k+ reputation share the data with you here on a public forum like MSE allows everyone to (at least partially) investigate this for themselves. Ain't the open and transparent sharing of information great? (And, yes, unlike the moderator-only analytics, there is no restriction on sharing this data publicly; in fact, it's explicitly encouraged to share it on Meta, from a time when the company valued that transparency and openness that attracted us in the first place.)

Anyway…Looking at aggregated up+down votes from January 1, 2023 to today (June 13, 2023), here's what we get:

Plot of total up and downvotes per day on Stack Overflow.

Before anyone complains about the scales, these are what the built-in tools give you. I agree it would be better if the Y axis started at 0, not a value slightly below 15k. I agree that the X axis is weirdly labeled. But it's still clear enough what is happening here:

  • The total number of votes has been trending downward since about February 2023;
  • A minor but noticeable dip in the voting rate is observable at the beginning of April 2023;
  • Voting rates were roughly stable from April to mid-May 2023;
  • There is another minor but noticeable dip in the voting rate observable at the end of May 2023 (it can be reasonably concluded that this aligns with the introduction of the restyled vote buttons).

It's not a huge drop; perhaps it's not even a statistically significant decrease. But the voting rate is definitely not showing signs of improvement, and there is nowhere near a 28% improvement.

The various features are even more noticeable when we look at a 7-day (weekly) average of voting rates over the same time period:

Plot of total up and downvotes per week (7-day average) on Stack Overflow.

I'll be honest, this complete lack of noticeable increase in the voting rate surprises even me. I was always skeptical of the claim that redesigning the vote buttons would have any lasting effect on the total number of votes, but I was not at all surprised to see that the change in styling produced a temporary effect. Any time you change things, people are going to be curious about it, interacting with it more than normal. That's why I never really got excited about this change. Although I am a huge advocate of voting—both directions—and would support anything that increased it substantially, I never bought the argument that the classic vote arrows were insufficiently obvious or in any way to blame for the decline in voting. (How could something that has been unchanged for years be responsible for a decline, anyway?) It was easy enough to explain away the "data" that the company presented showing that this redesign increased voting as nothing more than a temporary boost arising directly from the fact that there was a change.

There is even historical precedent confirming this temporary boost in voting whenever the styling is changed: as the April Fools' Day prank in 2011, unicorn animations were added whenever a vote arrow was clicked. There were unicorns everywhere—and the people loved it! People were clicking vote arrows all over the place just to see the cute little unicorns! How much did it increase voting? Sam Saffron (aka "waffles"), a staff member at the time, provides the answer: a little over 5% for upvotes and nearly 14% for downvotes (awesome!). About half the 28%, for a much more impactful and interesting change, but April Fools' Day is a highly localized event, so it's reasonable to assume that there were many people who didn't visit the site that day who never even saw this change.

What if we look only at upvotes or downvotes? The trends are similar:

Plot of total upvotes per day on Stack Overflow.

Plot of total upvotes per week (7-day average) on Stack Overflow.

Plot of total downvotes per day on Stack Overflow.

Plot of total downvotes per week (7-day average) on Stack Overflow.

You can see the effects of the ongoing moderation strike prominently in the downvotes. There's a significant drop in the number of downvotes on June 5, 2023, coinciding with the start of the strike. The effect on upvotes is far less noticeable. (It should surprise no one that there are fewer downvotes when the most quality-minded of users go on strike, whereas there is little or no change in the upvoting rate.) Still, even if upvoting rates are remaining flat, they're not increasing.

One might argue that the strike skews the analysis of this data; I claim the opposite. Throw out all of the data from June 5 onwards during the strike, and you're left with a little less than a week's worth of data, from May 30 to June 4. This is literally during the "honeymoon" period, immediately after the roll-out of the change, when the above analysis suggests that voting should be skewed significantly upwards, only to level out again later. This change didn't even cause a notable temporary/transient uptick in the rate of upvotes.

As Richard sardonically pointed out, the answer to encouraging more voting does not seem to be "putting a big dumb ring around the voting buttons" (ironically, making the only meaningful semantic element, the triangles, much smaller, not to mention lowering the contrast of everything).

19
  • 2
    Did the number of visitors also go down over this period of time? If so, it would be interesting to see how it correlates with the decrease in votes. Maybe if the number of visitors dropped more than the number of votes, one could argue that the new buttons have a positive impact on the number of votes (although I wouldn't draw such conclusion without more data, since the visitors that are leaving the platform are probably not the ones that are the most interested in voting anyways).
    – Dada
    Jun 14, 2023 at 10:49
  • 2
    Analytics also show total number of new posts (questions plus answers) have the same downward trend: from about 70k/week in Jan. to about 50k/week now. I think it is reasonable to assume that fewer new Q & A mean fewer votes.
    – toolic
    Jun 14, 2023 at 10:49
  • +1. This is a comprehensive compilation of the data that provides insight to what we already might have guessed. The intention might be to increase the voting rate but redesigning the arrows always seemed ludicrous (especially how it was implemented). They could have incentivized the same by, maybe, introducing badges, privileges etc. What I mean is a more deep thought and analysis and more importantly providing a meaningful feedback interaction ecosystem to the community before overhauling something are important for the overall success. Jun 14, 2023 at 10:53
  • Thank you for this answer. It even seems possible that the new voting style hurts voting actually. Just to exclude that people simply didn't find anything to vote on since the button style changed, would it be possible to show the number of new questions+answers or the number of visitors as control? I believe nothing much changed there around May 30. Jun 14, 2023 at 11:04
  • 8
    Is there a similar downward trend after May for prior years? That’s the start of summer break and when most college students graduate in the US. I think your analysis indicating the change didn’t boost voting is spot on, I’m just curious whether the drop is seasonal. On ELL, the question volume definitely waxed and waned with the school schedules, but its audience is a lot different from SO.
    – ColleenV
    Jun 14, 2023 at 11:15
  • Another confounding factor is the poor initial design, where many users were unable to tell whether they had already voted or not. I still find myself losing votes where I know I had clicked to vote on something, and also voting again after a page reload where somehow apparently the web UI dropped the status of my vote.
    – tripleee
    Jun 14, 2023 at 11:31
  • 1
    @ColleenV Not really; here's a chart from Jan 1st 2015 to today (the X-axis is labelled oddly, but it really starts in 2015). The dips are around Christmas, and it seems there's been a steady decline since roughly late 2020 after a peak in mid-2020 (probably COVID lockdown related). This also matches declining question rates, the trend of which has been downwards for a while. Jun 14, 2023 at 11:33
  • 5
    For the purpose of this question we should probably be looking at "avg vote per post" rather than total number of votes. In the week of May 3 2020 there were 101,145 posts and 314,913 votes, in the week of March 13 2023 68,488 posts and 236,569 votes, and in the week of Jun 5 2023 50,062 posts and 181,908 votes. This corresponds to 3.11, 3.45, and 3.63 votes/post. So it does seem that voting is increasing, even if the total number of votes are dropping. Someone with more time can probably made a nice SEDE query for this. Jun 14, 2023 at 11:34
  • 13
    I don't think we can conclude from this that the new design hurts voting, since, as others have rightly pointed out in the comments, there are many other confounding variables (most notably, total number of posts decreasing). I do not have the time, nor the expertise, to do a full analysis. However, I think this serves as sufficient evidence, feeble as it may be, that the redesigned vote buttons are not leading to a significant increasing in user voting, certainly nowhere near the 28% increase that was postulated (and used as the basis for driving this through). Jun 14, 2023 at 11:46
  • 5
    The bigger problem here is just the complete lack of engagement, or any data to back up the assertion. It's like they're not even trying to convince anyone. My own gripe is mostly about the lack of contrast and I'm pretty sure a few years ago my feedback would have gotten some response. "We disagree, [status-rejected]" would be fine – reasonable people can disagree, and at least I'd know people are listening to what's being said. But by not engaging at all it's just "we don't even care what you have to say, sod off" and that's so much worse. Jun 14, 2023 at 12:27
  • 2
    @MartinTournoij But they don't look more like buttons. There's no drop shadow or other styling that makes them look any more clickable than the original design. SE has a bad case of big-code-base-incrementalitis. The only cure for a fear of making significant changes to a code base is comprehensive automated testing, a solid understanding of your user's needs and in some cases, preventing upper management from wandering into the Bog of Detail Outside Your Expertise.
    – ColleenV
    Jun 14, 2023 at 13:05
  • 2
    The circle does give more of a hint, I think @ColleenV. But I agree it's not a perfect design – these "flat" UIs are all the rage these days and the old buttons were also "flat". Also agree about "big-code-base-incrementalitis" etc. Jun 14, 2023 at 13:12
  • 1
    Yea but your data doesn't include attempted votes by low rep/anonymous users!
    – Kevin B
    Jun 14, 2023 at 14:49
  • 1
    Like Kevin, I'd also be curious about the attempted vote numbers by unprivileged users; I don't know if the analytics panel shows that, but I know SEDE does. Obviously that count doesn't actually affect voting now, but it might still be a factor for the Company in some way.
    – zcoop98
    Jun 14, 2023 at 15:58
  • 1
    @User1865345 - solidarity Mods the dev effort on those options are substantially different.
    – Piper Staff
    Jun 14, 2023 at 16:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .