How does Stack Exchange work around the lack of Redis clustering?

Question

I read that Stack Exchange uses Redis for caching and Redis doesn't have clustering support yet. So I was just wondering how Stack exchange works around this lack of an important feature? Does it use servers with huge memory? If so, how much? Or else, what is your approach?

Answer 1

Basically, yes: we are currently simply inside the memory limit of the servers. We haven't intentionally gone for "huge memory", but they do have an amount appropriate to their role. We only solve things that are actually problems, but we have plenty of options. To explain our setup - we currently have only 4 physical redis servers for the production environment (servers for dev/QA/etc are irrelevant to this discussion); 2 in each DC. We have 8 redis instances on each redis box currently, that each serve fundamentally different roles - one for our core cache etc (where "core" here means "not careers"), one for machine-learning, one for pub/sub distribution, etc. These 8 instances are then replicated (master/slave) between the servers for HA / DR, persistence and performance reasons (noting that SE.Redis allows individual commands to be targeted, i.e. "demand master", "prefer master", "prefer slave", "demand slave").

Currently (glances at our opserver monitor) we are running at (picking a server at random), for the 8 instances: 28.55GB + 4.51GB + 1.66GB + 15.18GB + 2.55GB + 1.72GB + 224.79MB + 82.26MB

The OS reports 56.71GB of 94.48GB memory used (60.04%) - so: some headroom, but if we exhaust our available memory, we have plenty of options:

• add more memory
• move some of the roles to different hardware (in particular, get the 28.55GB and 15.18GB roles onto separate hardware, and juggle the rest to suit)
• proxy-based sharding via something like twemproxy (SE.Redis supports this already, but I have not investigated whether all of the operations we use are provided)
• client-based sharding - not currently implemented (except for "cluster") in SE.Redis, but all the hooks and logic are in place (because: "cluster") - it would simply be a job of implementing one or more sharding conventions in SE.Redis
• server-based sharding (aka "cluster") - the code exists in SE.Redis, but "cluster" is not yet RTM, and there are some issues, in particular relating to multi-DC replication and promotion; this is discussed more here
• caller-based sharding - which could be useful if we wanted to move individual databases to different physical or logical servers

---

In the comments, @bitoiu requested more information about DC failure, also mentioning cluster support. On the topic of redis-cluster, our custom (and publicly available) redis client has preliminary support for redis-cluster (although it needs some more work), however, at the current time:

• redis-cluster is still premature
• redis-cluster lacks a good multi-DC story

Salvatore is aware of our needs on the second one, and has some things on the horizon to address them; as a one-line problem statement, it would probably boil down to:

if a NY (DC1) node fails, do not ever automatically promote an Oregon (DC2) node to take over as master for those slots; if we want to fail to Oregon, we don't mind pushing a big red button to do that explicitly.

So for now, we're using regular redis replication between DCs; the NY web-servers only know about the NY redis nodes; the Oregon web-servers only know about the Oregon redis nodes. That means that for most things, Oregon discovers that redis is entirely readonly (except for a set of nodes that is DC-specific - replication inside the DC, but not between DCs). In the case of DC failover, we simply make one of the other-DC's redis nodes "master" - ideally using a tool that broadcasts a "please recheck the redis configuration" at the same time (and conveniently, "opserver" is one such tool).

Answer 2

It's unfortunate that the documentation for Redis is misleading - and recent mention of upcoming clustering support also contributes to this misunderstanding. Redis fully supports "clustering" the way that many people are thinking, which relates to high availability and fault tolerance. You can have multiple Redis instances running, one is the master, and you have one or more slaves. All instances can service read requests, but only the master can take write requests (although that can actually be configured such that slaves accept writes - I don't understand that use case). If the master goes down, there is another Redis service called Sentinel that can be used to automatically promote one of the slaves to become the master. When the previous master comes back up, Sentinel reconfigures it to be a slave.

The functionality that is not RTM yet, which is referred to in the Redis documentation & release notes as "clustering" is in fact the ability to partition the storage of keys across multiple Redis instances. It may have been better perhaps had they decided to call this functionality "sharding" or "partitioning" instead of clustering.

If you have a need to have a Redis database that is so large that it would be difficult to host on one machine and have it be fully in memory, then yes, you're going to have to wait for the functionality that hasn't been released yet. But if your data set easily fits in memory, and you just want horizontal scalability to correspond to the horizontal scalability of your cache clients, then Redis is more than capable of working for you.