Winter Bash 2014 post-mortem

Question

I left out a lot of detail in my recent Winter Bash 2014 wrapup because it was too "inside baseball" for the general public. Before I forget, I thought I should put my post-mortem to metapost.

Things that worked

• Start early—I got roped in involved with planning WB 2014 on November 6 and Laura had started work earlier than that. Even so, there were details being finalized right up to the start of the event. I wouldn't say it's never too early (because I was pretty sick of WB even before it started), but having essentially a month (modulo Thanksgiving) was luxurious.

• Reuse and extend—At this point we have a solid framework for this event, so we can reuse a lot of the code and copy. One of my goals was to have all new hat ideas and triggers, but that turned out to be too ambitious. So we had mostly new hat triggers, all new hat illustrations (though Eureka! and the pirate hat concepts were carried over from last year), and a new theme for the Winter Bash page. Hat scaling and rotation was the main new feature that balpha added on the code side.

• Think about what you can learn—The Red Baron trigger is a great example of something we hoped to learn more about. There were several other hat ideas that were specifically included to find out how people would respond. I haven't done the analysis yet, so I don't know if we got useful results. But I think attempting to learn something is almost always a good idea.

• Sites opted in by default—In the past, we gave site moderators a choice to participate with the default being "opt-out". If a mod team didn't get back to us, their site was left out. This year, the default was "opt-in". Thanks to the default effect, we only had 2 sites opt out. Even though they didn't need to, we got 87 responses from moderators to opt in. I also got a number of emails from moderators expressing their excitement about the event. Besides getting more sites involved, this also made tracking of moderator responses much easier.

• Hat ideas from the entire company—Since I spent so much time refining and thinking about hats leading up to the event, I feel like each of them was mine in some sense. But the truth is the majority of hat ideas came from other Stack Exchange employees. It was really wonderful to see the brainstorming spreadsheet fill up with such inventive and imaginative ideas. We did need to skip dozens of fun (and funny) suggestions because of technical and social reasons, but that still left many ideas that I can't claim credit for.

Things that could have been better

• Typos and insensitivity—It's weird: none of the controversial hats were the ones I predicted would cause problems. Not much else to say except that several co-workers talked me out of a hat (already illustrated and ready to go) that would have been way more offensive than anything we actually used.

• Eureka!—On the one hand, I feel like we were fair and generous about who got this hat. But on the other, it probably doesn't seem like we were.

  Just before the event, I sent around an email to all the people able to give out the lightbulb with criteria and things to look for. (For instance, if you didn't have one of the hats triggered by voting, it was unlikely you'd figured out how they work.) We tracked guesses around the network and verified that discoveries were probably independent. We gave partial credit and co-credit when people worked together. But all of that scrutiny needed to remain secret in order to keep the hat triggers secret. Even if we got it 100% right (and I know of at least one person who missed the hat because we noticed the discovery too late), it would still seem arbitrary and like we played favorites.

  I'd personally be glad to ditch it next year since it just added stress for me. But I also like the idea of rewarding people for applying the scientific method to solve our puzzles. So I'm torn.

• I wimped out on Secret hats—My initial draft included several secret hats that were downright diabolical. I got worried that they would be too tricky to guess, so I dialed down the difficulty a bit. Some hats became non-secret and others were given easier-to-guess triggers. As a result, all the secrets were (more or less) guessed in the first two days. We need trickier secret hats next year.

• Fewer people than expected participated—Stack Exchange saw phenomenal growth in 2014. But that wasn't reflected in Winter Bash 2014 participation. Compared to last year, only about 10% more people earned hats. There are lots of possible reasons:

  • people got bored of the event
  • hats were harder on the whole to earn
  • fewer hats were available

  I suspect that the majority of people earning hats got them by doing ordinary stuff on a site rather than by setting out to earn them. If so, the reason for less-than-expected participation has more to do with the frequency of triggering actions (and fewer hats) than anything more worrying.

Thank you all, for a pleasurable and successful Winter Bash 2014. I look forward to getting excited about hats again in November.

If you have any ideas for next year please post them as answers to Winter Bash 2014 closing remarks - ideas for next year. On the other hand, if you have critiques (that aren't already raised on meta), stick 'em here.

Answer 1

I wimped out on Secret hats—My initial draft included several secret hats that were downright diabolical. I got worried that they would be too tricky to guess, so I dialed down the difficulty a bit.

This was exacerbated by Eureka; not only do people want to guess and be correct, but they want to do it in public so they get the reward. I suggest that removing Eureka (and thus the incentive to share the secrets you've learned) will help keep the hats more secret to begin with, and once you're not rewarding the guesses then who cares if you throw in a couple that are truly hard? We get a puzzle to solve, and you'll reveal it in the end anyway.