A/B testing the sticky vote controls [closed]

Question

First, apologies for not creating this feedback thread earlier - in hindsight, a change to such an established feature of our sites really does warrant some explanation.

We're running an A/B test on Stack Overflow only that places half of registered users with >= 15 reputation in a variant group that will see voting controls like so (shamelessly stolen from ᔕᖺᘎᕊ's answer here):

---

---

A few points about this current experiment:

• If you're in the variant group, the controls will only move if you can vote on the post; your own posts will not have the sticky controls
• This will run for another three weekdays, so it should finish up next Wednesday, 2015-06-10.
• We've set acceptance criteria at 5% more voting than the baseline.
• After the experiment concludes, I'll update this question with the results.

Edit 2015-06-08 01:50

We're into the next variant with these changes, based on feedback:

• stickiness is applied if the post's body is longer than 2/3 of the browser window's viewport
• stickiness is applied regardless of a post's votability, e.g. the controls will be sticky on your own posts (if they're long enough)

Edit 2015-06-09 19:55

Experiment has completed; preparing a summary.

Edit 2015-06-12 18:30

TL;DR this feature will not be implemented, as it had too little of an impact on voting.

We ran two* variants of these sticky voting controls:

1. variant 1 results: movement was only enabled on posts you could vote on; no minimum height requirement to the post body; ~3.8% vote loss from baseline
2. variant 2 results: movement was enabled on all posts if you had at least 15 rep; post body height had to be longer than 2/3 of the viewport; ~2.1% vote gain from baseline

Some definitions for the above result links:

• baseline: the current behavior, i.e. voting controls do not move from their initial placement
• trial: a question page was rendered to a user who could at least upvote something on the page
• success: the user upvoted or downvoted one of the question or answers

We placed a 5% minimum improvement threshold for acceptance; this is completely arbitrary, but it's a common floor in our experiments, as new features == new upkeep. Also, judging from the extreme polarization of the responses to this feature, we would need a user preference to disable it, something we're hesitant to provide for any feature.

Going forward, we might try another experiment with duplicating the controls at the bottom of long posts, but there's no ETA for this.

If you did like the functionality, it can be added via ᔕᖺᘎᕊ's Stack Overflow Extras (SOX) script.

_{* an initial variant was stopped after only an hour and this change was applied}

Answer 1

I don't like how it's following me. If it does get deployed, I hope there's a setting to turn it off.

Answer 2

Very very nice feature which came from the community and was instantly implemented. My compliments for the speed in which it was implemented.

The feature in general works fine. I still have to get used to it a little and sometimes its current working isn't really intuitive.

Besides that: a lovely feature!

I wonder though why there is a baseline of 5%. Do you expect more users to vote? Why not make it easier to vote and see what happens then? Even if no one votes more than they do now, this is a useful feature to me.

Answer 3

I think it feels weird that the controls don't move on my own posts. My initial reaction was that it was a bug, and I had to look it up on meta to find out that this was intended.

Even though I can't vote on my own posts, it's still useful to see my own score and it feels kind of jarring when suddenly one post is behaving differently.

Answer 4

I like this because I far too often forget to vote on lengthy posts because the buttons escape my view. I want to wait for the test to finish to say for sure how much this is actually needed, but I suspect the impact is going to be far from marginal.

That said, two things I'd like which would make this more awesome:

Don't scroll when the post length is less than or equal to the height of the voting controls.

I don't know how expensive that's going to be server-side since it would need to be known in advance, getting this from the client would be icky if even really possible. data-postlen=?

Dim the controls when scrolling comes to rest for a second so folks can read, restore opacity when scrolling resumes

I'd like to see how this alleviates some of the "crazy ass things are following me, man!" complaints that folks have, and might alleviate some of the distraction others have reported. They're made more subtle as you read, more prominent when the rest of the screen is moving.

---

I want to see the numbers on how much more voting we're seeing. I've personally found this useful several times today, but actual numbers are going to be interesting.

Alternate

Put up/down arrows at the bottom of posts, similar to how the links to close / flag are oriented (in fact, before them) - but again, I want to see how this test pans out.

Answer 5

I think the biggest thing I dislike about it is that when I am reading a post, something is moving next to me and is distracting.

Especially for longer posts this can be a bit annoying as they normally require more concentration.

That being said...

This will run for another three weekdays, so it should finish up next Wednesday, 2015-06-10.

There will always be resistance to changes, perhaps consider running this a bit longer than just a few weekdays to get a better feel (unless preliminary voting numbers make it clear immediately).

Answer 6

I've provided feedback in a separate post but I'll move it here to keep everything tidy.

---

This is related to the previous feature of not moving vote buttons past the bottom of the post.

I'd like to request that the "bottom" of a post be redefined to be the bottom of the text and not include the user card. On short posts, the user card introduces unnecessary scrolling.

---

For example:

This answer is smaller than the size of the vote/score buttons:

If, however, I scroll down to start reading the other answers, the buttons follow just enough to be distracted by the moving UI component. It follows to the bottom of the user card.

Notice that the buttons on the above image followed me down to the "add comment" (just below the user card).

The stickiness is nice on long posts (like the question portion of the link above), but several of the answers are short and the stickiness is distracting when it occurs unnecessarily. I think this can be resolved by only keeping the buttons sticky within the size of the answer text and not including the user card.

Answer 7

I think StackExchange is great regardless of this feature (I am neither against it, nor in favour of it), but if you are going to implement it, at least consider fixing this bug.

On my smartphone (in my case an iPhone 5c, running iOS 8.3, latest as of June 6th 2015 - but I think this is related to screen size rather than specific device) the vote buttons are moving over the post when zoomed in. This is bad, because it makes both the vote count and - perhaps more importantly - the post itself hard to read.

Image of bug (top status bar cut off):

Edit: I forgot to include the solution I would propose: let the vote count go off-screen (i.e. only change y position (relative to the page, not the screen) and preserve the x position (also relative to the page).

If another solution is found preferable, I'm fine with that too. I proposed a solution only because I think it's better to say "I don't like X, maybe Y is a solution" than to just state "grrr I hate X".

Answer 8

How are you going to analyze the results of the A/B testing?

One problem that may arise is that people will be voting by accident on the wrong answer, because two sets of controls will be right next to each other.

Some of them will notice the mistake, un-vote, and then vote again on the correct answer. This is something that you can track without much effort. But some of the people won't notice it, and this is something that you won't be able to track.

Answer 9

I am apparently part of the split/test of the new widgets and I was directed here from my post on meta.stackoverflow regarding the moving vote widget. Moving animated vote counter widget, can animation be configured off?

Here is the sum and substance of the question/discussion regarding the widget and a suggestion that there should be some way for users to disable the moving object on a per-user-basis. Some may love them, others may want to pick up a hammer and permanently fix it. (a configuration option in between is probably a happy medium):

In the past day or two, the little vote widget for questions has started following the question up and down the window for the extent of the question. If it is a long question, that little widget becomes quite annoying.

While I'm all for improvement, I caution against adding all the latest 'gee-whiz' animated bells and whistles just because you can. SO is clean and non-intrusive. The more animated widgets get added, the closer the site moves to the dividing line between clean/enjoyable and annoying/distracting. The line is different for all users, and I'll admit that I prefer a clean usable interface without animated distractions.

This isn't a complaint saying remove it, but rather a suggestion that animated parts of the site have some type of user-preference that allows users who find the moving widgets annoying to simply -- turn them off. (or disable moving animations, or something similar)

The script features that update your comment inbox, etc. are all perfectly fine, they provide no distraction because they don't move. Having the vote counter flittering up/down the left side of the browser window is something I would like to turn off. Recognizing others may love it and think it is the greatest thing since sliced-bread, don't de-animate it, but please find a way to allow users to turn it off.

Remember, somebody's latest gee-whiz feature, is a bug to someone else -- if it can't be turned off. In the case of the flittering widget, on questions with long answers, it is horribly annoying to have the little controls racing down the screen as I scroll down.

Answer 10

I'm one of the lucky ones that have the new button behavior. And I don't like it.

At first I didn't know why I should care if this thingie moves around.
But I now realize that I have used the voting buttons as an highly visible divider between question and answer, and multiple answers.

When I read long questions I often scroll back and forth multiple times. Since I scroll in a quick manner, the voting indicators were a great way to know when I scrolled too far. That no longer works.

The voting indicators where also a great way to know when I scrolled to the top of an answer. Now I have to scroll to the previous answer, search for the tiny hair line between answers, or I have to observe when the voting buttons stop moving.

Once that thing is live, I will do whatever it takes to get rid of it. So please don't make that too hard.

Answer 11

Animation on websites annoys/angers me immensely.

If i want animation, i go to the TV or to YouTube.

If i want information, i go to sites (that don't have animation).

Answer 12

For those “lucky” enough to be testing this, here is how you get rid of it. Note I am using Firefox with Stylish. When you scroll a second div is created, and if you stop the scrolling on some answers this second div creates a space between the answer and the comments.

.js-vote-sticky {
  position: static !important;
}
.js-vote-sticky + div {
  display: none !important;
}

Source

Answer 13

I've been put in the group with the sticky vote controls.

I've learned today that I rely on those vote controls to indicate the structure of the page to me. In particular they indicate the top of the question and each answer. They're like the bullet points on a list, on a page that is just a really big list. If they're moving around, I lose the structure I depend on to make sense of things.

• If I'm busy scrolling, I no longer have a sense of how many answers I've been past and where on the page they are located.

• If I stop scrolling in the middle of a large answer and I'd like to read it, my ability to locate the top of that answer is currently impaired. I usually scroll up until I see the voting controls. Instead I scroll up until they stop moving, and then re-orient. If I'm scrolling too fast, I might confuse those voting controls with those of the previous answer for a moment before realising my mistake. (They're right there at the top left where I'm focusing, after all.)

It's actually surprising how disorienting it is. The pages lose the key dependable structural element I have apparently learned to depend on heavily. And I scroll and skim fast through answers, so this is a big impairment for me.

Call it adjustment, certainly. I've built a mental model for rapidly navigating Q&A pages and I'll have to rebuild it. But that is going to take some serious adjustment.

I would prefer not to have sticky vote controls. I like my bullet-point vote controls.

Answer 14

I don't mind it scrolling vertically, although it was a bit unsettling at first. But please don't scroll horizontally! I browse with a large text size, so sometimes need to scroll horizontally, and the vote buttons follow me across the screen, obscuring whatever text of the question/answer is in the top-left corner.

Answer 15

I'm finding it disorienting.

The "vote" buttons behave "unpredictably", visually.

As I'm scrolling down, some of them are moving "up", but one is staying still at the top of the page, until suddenly it pops off the top, to be replaced by another.

No, thanks.

Answer 16

I find this extremely annoying. I don't want to see it. Ever. Not only do I not care about the score of a post while I'm reading it I think it begs for votes way too much. Not to mention being terribly distracting.

If I want to vote I can scroll up the page some. I think a much better solution would be to add mostly transparent "go to top of answer" buttons at the bottom of really long posts. Or, if you could do it unobtrusively, detect if I am rapidly scrolling and add both bottom and top buttons next to the text (again, mostly transparent so it isn't as jarring), but I'm not sure if that would have the same terrible annoyance of the current proposal.

Answer 17

I agree about the "not scrolling vote controls on your own posts," but I can't change it without invalidating the current experiment. Based on the results, we might run another test where the controls always move. – Jarrod Dixon

Evidently I'm not in the variant group (I'm not seeing any change) but I support the original suggestion that we locate the voting buttons in a fixed position at the foot of the post. I'd like to see the score alongside them, but also remain at the top of the post (without necessarily having voting buttons there). I suggest this arrangement for two reasons:

• It makes sense to have actually read the post before voting, while
• It also remains useful to be aware of how others have voted before reading.

From a quick scan of remarks by others here and elsewhere, I see I'm not alone in being not at all keen on the idea of things moving around.

So, with the comment I've quoted from @Jarrod Dixon in mind, can we please also have a test where the controls never move? And a vote on the final choice about any movement, after we've all had a chance to try out the different ways?

Answer 18

I don't care actually. If answer is good and helped me — I vote it up, regardless where these buttons are.

Answer 19

Works on this, https://stackoverflow.com/q/30673961/1927206

Does it need to move around on posts of that size? I was confused. If it had been a post which went off the end of the page, so the voting mechanism needed to move at some point, I'd have probably worked out the purpose.

There are probably going to be at least two camps on this. My camp is "only move when needed", by the length of post (if possible, I have no clue) and ability to vote.

If you are analysing whether this generates more votes than otherwise, I think you should separate the "short" posts from the "long" ones. You'd not really expect more votes when the movement wasn't needed, presumably the intent is to get more votes for longer posts by making it (physically) easier to vote on those?

Are there really enough "long" posts to have an impact?

And, is it my imagination, or does the voting scroll into the comments in the mock-up in the question? That would be bad.

That would be bad, why? Someone or someones are going to "vote" on a comment they like. Sometimes they'll realise (then, or at some later point). Sometimes they won't.

Answer 20

I think this feature is not bad per se, but weighting the pros and cons and the main goal, I would prefer an option like the alternate in TimPost's answer or the first part of LadybugKiller's answer here.

Something moving (or animated) distracts me a lot and avoids me to concentrate in what I'm reading.

As the main goal is to increase the amount of votes (up or down) while doing it easier (without scrolling up and down after every answer), that could be achieved without moving the vote controls:

1- If the answer if short enough, I don't need to scroll (I still see this control on my screen), so it isn't needed to move it.

2- If the answer is long enough, a repeated control in the lower part would do the job without any harm.

If you want not a repeated control, just show it at the lower part of every answer, as it would be very uncommon to vote an answer while reading just the upper part.

Answer 21

Could the distance the votes stay from the top of my screen be the same as the distance between the vote and grey seperator when it's not scrolled?

This is the distance I would like it to be - from the faint grey line to the point of the upvote. I make it 50 px from the line to the point:

But closer to 70 from the window top to the point:

Please can these be the same?

Answer 22

I think it'll be cool if deployed for Desktop Users. It must be added as an additional feature. It can be a bit annoying for Mobile Users as I too use SO and other sites through my phone and the floating vote button will hinder my experience on the small screen size.

When thinking on a large scale, I think there would be plenty of users who uses SO through their mobile and tablet devices with each one having a different screen size. So, even if you introduce it, you'll be playing around with live users. I think SO is just fine as it is now.

Answer 23

It would make sense to put duplicate vote/accept buttons at the bottom of a very long post, so you don't have to scroll several screens back up to vote. Instead of them moving, of course.

Answer 24

I am not sure how much effort it would be to implement this conditionally, but Firefox natively supports this via CSS. Something like the below CSS would be able to replace the JQuery library and the JS code in Firefox.

.vote {
  position : sticky;
  top : 0;
}

So why not simply let the browser handle all the processing and code to get the same behavior?

Answer 25

Thank you for not making it "smooth scroll" like most auto scrolling advertisements are. It looks like a nice feature, but for small and simple posts (like this answer will likely be), it feels very weird to have a small element remain fixed and then move instantly.

Can we have it turned off if the rendered length of the post is like half of the screen or less?

Answer 26

Scrolling vote buttons make it much easier to read the first (long) answer, vote, and then continue to other answers.

This would be especially useful on sites like Worldbuilding, where many answers have pieces of the complete solution. I find myself scrolling up a whole screen to vote, then back down to read the next answer.

Answer 27

I like the stickiness; it's convenient and I harbor none of the "omg, it's following me" reservations raised by other users. That said, I'm not too keen on this particular line-item:

• stickiness is applied if the post's body is longer than 2/3 of the browser window's viewport

What problem was this attempting to address? All it seems to do is introduce inconsistent behavior.

Answer 28

I haven't gotten a chance to try this, it seems like it could be useful, but instead of following why not just permanently fix all vote buttons to the bottom of the post instead of the top? You're expected to read the post anyway before voting (at least I think you are), so it seems to address the "I don't feel like scrolling up" premise without the stickiness that some find irritating. On the other hand it's useful to have the score at the top so maybe this comment answer isn't the greatest idea. But people whine about score affecting votes anyways, so maybe score fixed to top, buttons bottom...

I think this would be a simpler way to address the identified issue, plus some.

Answer 29

I noticed this yesterday, while working on an answer, including that my own answer votes arrows would be fixed. It looked nice.

I cannot reproduce this functionality anymore, it is kind of confusing to not be permanently in the variant group.

Answer 30

Why not make it an option? As for me, I'll grab it immediately!