27

With so many StackExchange.com sites, in addition to StackOverflow.com, I was curious which approach is taken for the multiple sites:

  1. One deployment environment (data/site/etc.) that is multi-tenant and serving a differently styled site depending on the URL used to get to the site, or
  2. A separate deployment environment for each site.

What were the most important decision factors for making the choice?

  • This seems to combine two questions, one that belongs here and one that doesn't: (a) what does this network do (b) what factors should a site use to determine what they should do. – Aaron Bertrand Oct 2 '13 at 1:06
  • (2) is almost certainly true, unless there have been major changes in the last year or so. – waiwai933 Oct 2 '13 at 2:17
  • @Qantas94Heavy: Most of the differences between sites can be attributed to individual CSS files and software settings. – Robert Harvey Oct 2 '13 at 5:43
  • @waiwai933 (2) has never ever been true. – balpha Oct 2 '13 at 6:12
  • @Qantas94Heavy sorry, but that is entirely incorrect – Marc Gravell Oct 2 '13 at 7:02
  • @MarcGravell: always good to have correct information, thanks. – Qantas 94 Heavy Oct 2 '13 at 7:25
  • Note to self: if you want to get responses from SE developers, ask a question about architecture. Two posts in 30 minutes, impressive. – jmac Oct 2 '13 at 7:28
  • @balpha I was under the impression that although the SE sites were on the same server (MSO and previously SO excluded), they all ran separate instances. Is that not true (or rather, was that not true at one point)? – waiwai933 Oct 2 '13 at 7:29
  • 2
    @waiwai933 It looks like "never" was incorrect; I just checked the repository history, and it looks like the trilogy started out as one-app-per-site. But it definitely hasn't been true ever since StackExchange 2.0 arrived about 3.5 years ago. – balpha Oct 2 '13 at 7:45
23

1, basically. The only site that isn't in the same space is meta.stackoverflow.com meta.stackexchange.com, which has a separate deployment to allow us to use it as a test-bed. It is preferable to find any show-stopping configuration problems (etc) on meta, than to take the entire network offline.

Reasons:

  • simplicity: we don't need to keep adding new sites in IIS, deployment tools, etc - just a single wildcard map and we're done
  • performance: the infrastructure and code only needs to spin up once per server for the entire network - not once per site per server; this is especially important for the lower throughput sites to avoid cold-start penalties (and while each site will start with an empty cache, that is quickly reloaded from redis without hurting the underlying data-store)
  • memory: a lot of the memory space is going to be identical between sites - they are, ultimately, loading the same dlls, compiling the same regexes, caching the same pieces of global state - better to load it once only and share it between sites
  • shared resources: every site will need to talk to some key services - for example, while we have per-application databases, we also have databases and redis nodes that are shared between all sites; for these, it is preferable to reduce resource overhead by having O(servers) connections, not O(servers × sites) connections

In some hosting scenarios, there might be privacy issues with two unrelated customers very paranoid about sharing process space with a competitor (or just someone malign) - but since we are only co-hosting with ourself, this is not a concern for us.


In the comments there is some mention of per-site differences; it is true that each site has some different options, but there aren't any deployment difference between sites - everything is handled by a bespoke flexible configuration system drawn from a shared database, which allows arbitrary settings to be set at various levels:

  • a code default
  • a network-wide default
  • a site-specific value

So basically, a lot of code will just do things like:

if(Current.Site.Settings.Security.GlobalAuthEnabled)
{
    ...
}

or

if(tagLength > Current.Site.Settings.Questions.MaxTagLength)
{
    ...
}

where Current.Site.Settings simply resolves the appropriate settings object based on the desired host in the request. A lot of these values would otherwise need to be magic numbers anyway - for example, "how many points is an upvote", "how many votes do you get a day" - so this implementation solves both problems in a single unified way, that can be reconfigured trivially on the fly.

Differences like images and css are handled simply by resolving the correct files based on the request host.

  • Hmm, maybe we should coordinate answering these meta question ;) – balpha Oct 2 '13 at 6:55
  • @balpha heh indeed ;p – Marc Gravell Oct 2 '13 at 6:56
  • @balpha you really should write a script showing who's viewing/answering the question you currently view and have it blink if other dev is viewing. ;) – ShaWiz Oct 2 '13 at 6:58
  • @ShaWizDowArd the scary thing is : with web-sockets, that would be pretty trivial to do... – Marc Gravell Oct 2 '13 at 7:01
  • 3
    @balpha the reassuring thing is: at least we came up with basically the same answer; if our answers were strikingly different there would be reason to be concerned – Marc Gravell Oct 2 '13 at 7:04
  • @Marc I recall a dev (Waffles maybe?) mentioning there is already special dev-only panel showing stuff like loading time of each page and raw SQL it triggered... so you also got a place to add that! – ShaWiz Oct 2 '13 at 7:12
  • Huh, I thought changing the maximum tag length was pretty much impossible? Is that just an example that does not work, or would it actually be as easy as changing a per-site setting? – Mad Scientist Oct 2 '13 at 7:15
  • @ShaWizDowArd that would be mini-profiler, and yes it is awesome (it is freely available, btw) - but it wouldn't help here - indeed, the very fact that it is hidden away would make it unusable here - but for info: i.stack.imgur.com/NfvUn.png – Marc Gravell Oct 2 '13 at 7:15
  • @MarcGravell: Not sure you should be showing us a profile with duplicate sql statements! :P It might destroy the image of the SE devs being flawless! – George Duckett Oct 2 '13 at 7:17
  • @MadScientist that is indeed the same value on all sites currently; but we could make it smaller on a given site if we wanted; yes, making it bigger could be problematic – Marc Gravell Oct 2 '13 at 7:17
  • @GeorgeDuckett actually, that is a false-positive based on some changes we've done recently to show redis etc in there; the "duplicate" in there is actually where it did a "get" on "foo", didn't find a value, so then did a "set" on "foo". Not actually a duplicate. I'm actually fixing that in VS as we speak. – Marc Gravell Oct 2 '13 at 7:19
  • @MadScientist That's what Marc meant about "solv[ing] both problems" -- that particular example is not so much about being adjustable per-site, but about not having 25 appear in the code as a magic number. – balpha Oct 2 '13 at 7:26
  • @MarcGravell: Ahh ok, thanks for the info. (eagerly awaits MiniProfiler 3.0); 6-8 weeks. – George Duckett Oct 2 '13 at 7:29
  • @GeorgeDuckett should be very soon - I would hope less than that; and it is awesome i.stack.imgur.com/olWWg.png – Marc Gravell Oct 2 '13 at 8:07
  • 1
    @Sklivvz and Marc - thanks for the clarification there - love how open you guys are about this stuff. – RobVious Nov 11 '13 at 14:00
19

As far as web servers go, the answer is 1), with the exception of this very site.

When you request a page from any Stack Exchange site, this request will be answered by one of eleven web servers, NY-WEB01 through NY-WEB11. If your request goes to Meta.SO, the answer comes from WEB10 or WEB11; in all other cases, WEB1 thru WEB9 will answer.

This exact breakdown may change once in a while – until not too long ago, we had three groups, and whenever we fail over to our secondary datacenter in Oregon, things will obviously look different as well.

There is no technical reason for splitting Meta off; the only reason is that we sometimes like to deploy things to only a small portion of the users first, to make sure we didn't break anything. Meta is pretty good for that, because 1) it gets real-world levels of traffic and activity, 2) the regulars here are used to craziness, and 3) the Meta people are very good at identifying our newly introduced bugs pretty quickly.

But the Meta split is not mandated by the code; in fact, WEB10 and WEB11 have the very same setup as the rest of the web servers, and they would happily respond to requests for stackoverflow.com or bicycles.stackexchange.com – they just never get these requests, because the load balancers are configured to send only Meta traffic their way.

As for data, every site has its own dedicated database containing the site's questions, answers, users, elections, etc., and there's a single "master" database (called "Sites", although these days it contains much more than just that) that lets the code know what sites exists and what configuration settings to apply when responding to requests for different sites.

Almost all site differences are either purely CSS (each site has its own CSS file), or based on certain settings that can be turned on and off per-site (e.g whether MathJax or other plugins are enabled, whether the site requires the how-to-ask click-through, and so on). Based on the settings for the requested site, the code will include or not include the particular features in the response.

There are a few places where the code hardcodes something like

if (Current.Site.IsStackOverflow)
    //...
else if (Current.Site.IsChildMeta)
    //...
else
    //...

but this is the minority (and often only for historical reasons); in most cases feature distinction are made like this:

if (Current.Site.Settings.SitePlugins.CircuitLabEditorEnabled)
    // include the JavaScript you find when searching the
    // source of this page for "schematics.init"

As for the decision factors, I can't claim any first-hand knowledge (the first non-Stack Overflow site launched a year before I was hired), but considering that today our site count is in the three-figure range, it certainly was the right decision.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .