Stack Overflow has existed since 2008. At that time, it was a logical choice to run this from our own hardware in a datacenter. The company started with a small group of people who owned every aspect of the application, from the infrastructure to the code. We built a monolithic application that scaled incredibly well and we squeezed the most out of the hardware we had.
It was a great time.
But now, the time has come to leave our data center and move to the cloud.
Why are we moving to the cloud?
The tight control we had over our hardware really helped us build a scalable application that was cost-efficient. However, the decision to move to the cloud was instigated by three unrelated events:
Our data center in New York recently announced that it’s going to close. We would need to move all our hardware to a new location. Moving would be expensive, but most importantly, it would be an all-consuming project for 4-5 Site Reliability Engineers (SREs) for many months. That time could be better spent moving to the cloud.
Additionally, our hardware is reaching its end-of-support date and would need to be refreshed. This would be very expensive. Should we spend that money on continuing our old direction or use this as an opportunity to explore something new?
Lastly, maintaining our data centers was becoming a distraction. We estimate it would take 2-4 full-time people to properly maintain all our hardware. Sometimes hardware breaks, and this means our engineers have to physically go to the data center to fix things. Owning the hardware also means we have to do all the maintenance ourselves and sometimes upgrade the hardware when bits and pieces go out of support. This takes time and money from other things we want to do.
We’re already using the cloud for our Stack Overflow for Teams and Stack Overflow Enterprise products. Stack Overflow for Teams originally ran in the data center too, but a little over a year ago, we split Teams from stackoverflow.com and moved it to Microsoft Azure (you can read more about this journey here and here). Teams has since then moved from virtual machines to Kubernetes, and we started deploying independent microservices that are no longer part of the original monolith. Stack Overflow Enterprise has been running in Azure for a long time, giving each customer its own isolated infrastructure and scalability, and it, too, is moving to Kubernetes.
We love the cloud. People don’t have to go into the data center anymore (which is really handy given that we are a fully remote, international company). We also gain a lot of flexibility with our cloud usage. In the data center, we were constrained by how much hardware we wanted to buy and maintain. Cloud gives us flexibility to use hardware when we need it and stop paying for it once we’re done. We were already doing CI/CD on-premises, but in the cloud, we now also have Infrastructure as Code making it much easier to manage all our cloud resources. Add the Docker and Kubernetes foundation we’ve built for Teams, and we are in a pretty good place with our cloud usage.
Now we’re going to take ‘everything else’ to the cloud.
Everything else includes the full Stack Exchange network: stackoverflow.com, all Stack Exchange sites, all meta sites, all apps from Area51 and Chat to internal apps like StackMail (our email sending service), and Scheduler (which runs scheduled tasks, such as badge awards). This is a massive project that we’ve been working on for a while, and we want to finish by June 2025.
To make it explicit, however, we are not going to the cloud to save money. We know that cloud is often more expensive than running your own hardware. We are of course monitoring our cost and by right sizing our cloud usage, using tools such as Azure Reserved Instances and auto scaling our resources based on load we keep our costs to a minimum. However, the cost is worth the new flexibility. Instead of projects being delayed while new hardware was procured, installed, and configured, we can spin up new capacity in minutes. We can even set up capacity experimentally and throw it away afterwards. This should help us to get new features out of the door faster.
Where are we going?
As discussed, our Teams products run in Microsoft Azure. For the public platform, we’ve decided to move to Google Cloud. We have a tight partnership with Google and we think they are the right cloud provider for the public platform.
Who is working on the migration to the cloud?
The internal name for moving our public platform to Google Cloud Platform (GCP) is called Project Ascension. The Ascension team consists of several developers, site reliability engineers, our database team, and a community manager.
As you’ve probably guessed by now, I’m one of the developers working on this project. I’ve worked on the migration of Teams to Azure and it was a logical choice to bring that knowledge to Ascension.
One of the other developers is Steve Vakil. He came to Stack Overflow in 2022 after many years of software development work for government public assistance programs, is intimately familiar with our ads product, and is a Google Certified Cloud Engineer. The other two developers are balpha and Adam Lear who both have a rich history at Stack Overflow and know everything there is to know about our products and all the things we have to think about with such a massive project.
On the SRE side, we’re joined by Mike Frank, Jason Schwanz and Tom Limoncelli. They’re designing and building our cloud infrastructure which is no small feat considering the complexity of applications that are built with decades of assumptions about the environment they run in.
And, on the DBRE side, Aaron Bertrand, who was at the helm for migrating the databases supporting Stack Overflow for Teams to Azure, will be carving out the database infrastructure in GCP and planning and performing the migration of the entire network of databases out of our on-premises data centers.
What have we done so far?
At the end of 2023 we started planning Ascension and thinking about all the things we have to do. Early this year, the team slowly came together and we started the real work.
We now have built out the core of our infrastructure in GCP, such as Cloudflare support, networking, Kubernetes clusters, database virtual machines, security and policies. This is all built on Terraform and automated CI/CD pipelines. We then started deploying all the applications that are required to run stackoverflow.com into Kubernetes in our test environment.
Our biggest unknown was how our application would perform in a cloud environment. Our app was built within the constraints of the data center, where we knew exactly how much latency we had and what we could squeeze out of our servers. Could we make our application run in a cloud environment or do we need to make big architectural changes?
We are happy to say that we did make it work. We build a set of load tests using k6 and we slowly build up the load we can support by tweaking our infrastructure - from the right Kubernetes node sizes to nginx settings and SQL Server VMs. We now think we’ve learned everything we can from these synthetic load tests and we’re ready to move on to the next step.
What will be the next steps?
One step is already done: Stack Snippets is running in GCP without any issues. Stack Snippets is a relatively simple app that does not need a database or Redis. It helped us prove out that our infrastructure is ready. Using Stack Snippets we showed that we have the monitoring and observability in place to run applications in production in GCP.
Our next step is moving more complex applications, one by one, to GCP. We want to avoid doing a “big bang” where we migrate everything at once, since that introduces too many unknowns and too much risk. Over the coming months we will announce which applications we’re moving and move these one by one to GCP. Some of these announcements have taken place on Meta in separate posts, but we plan to consolidate them as an update on this post.
Once we have our internal applications all running in GCP, we plan on moving Stack Exchange sites to GCP. Our current plan is to start with some smaller sites that don’t take a lot of traffic so that we can do basic testing. After that, we will likely move meta.stackexchange.com to GCP so we can test performance with more traffic.
Once we’re confident that our GCP environment is stable and can sustain our traffic we will move our big sites, and we will end the migration with stackoverflow.com. While we'll do our best to minimize downtime, big changes like these frequently bring short periods of instability. We apologize in advance and appreciate your understanding.
Now of course this plan is very much in flux. We will learn more and more while we execute on it and things will change and we’ll try our best to keep you up to date.
I personally am really happy that we’re moving to the cloud. Having flexible infrastructure that’s deployed and managed automatically will give us a lot of opportunities to improve the way we work and build new features.
We are just at the beginning so we will keep you posted on what’s going on.
