I'm curious about Stack Exchange's HA setup for Redis. Based on existing blog posts and other answers, I understand that there are 2 nodes per instance per datacenter - a master and a slave. But how do you handle failovers? How do you deal with consistency issues when the old master comes back up? How do you prevent "split-brain" scenarios?
Firstly, it should be noted that unplanned failures (with that level of nodes) are exceptionally rare. The only one I can think of involves a certain UPS incident. We don't use "sentinel" - the likelihood and availablity (human and system) is such that in the event of catastrophic failure we can reasonably make the "call" ourselves at the time. And basically, we have a big red button we can use to promote a slave to a master, including setting some configuration data into a well-known key. When connecting, we try to talk to all the nodes we talk about, and we ask them who they think is in charge; if they agree: fine; if there are multiple masters and they disagree, then we use the data itself to pick a winner.
It helps a lot that we aren't absolutely dependent on the data anyway - often with SQL Server as the "trusted" back end store. If we were massively dependent on the data (and could afford zero risk of loss etc), then we would use different strategies.
Moving forward, a lot of the issues here are solved automatically by redis-cluster, which is something that we're looking forward to playing with more (as soon as the multi-DC features are there). Equally:
- because we know redis-cluster is on the horizon,
- and because we know that redis isn't causing us an active HA problem,
we're kinda "punting" on "solving" something that isn't actually a problem. We fix problems we have, with overkill - but we don't try fixing problems we don't have.