Meta Stack Exchange is a question and answer site for meta-discussion of the Stack Exchange family of Q&A websites. It only takes a minute to sign up.
Sign up to join this community
How - who does the backup? If a dedicated server, then where is it & how often does it backup?
Where - which place/server does the backup gets stored to?
When - at what intervals? How long does it take?
What level - what all content is actually backed up? How many layers of such backups exist?
Also, who has the authority to restore backups (in case one is required) ?
Has the site been restored before? (Related - Stack Exchange Blogs Compromised and Passwords Reset)
How - who does the backup? If a dedicated server, then where is it & how often does it backup?
So it's all automated of course, but Peter Grace (a.k.a. Router Hammer) handles the bulk of this work, taking care of getting everything to tape. We have a dedicated server called NY-BACK02 (we recently upgraded) which runs NetBackup inside our New York data center. It has a twin 10Gb trunk link to our network (so all backups are link local), an attached MD3200 DAS (via twin SAS) for on-site storage (and disk-to-disk-to-tape usage - 21TB usable), and is ultimately writing to a Quantum Scalar i40.
We do a full backup of all production SQL databases every night as well as transaction logs every 15 minutes. Most other things are nightly. We also perform full backups of SQL every night in Oregon (which is replicated near-real-time) for off-site storage (only to disk).
Where - which place/server does the backup gets stored to?
Backups end up on tape and on our backup servers (OR-FS01 & NY-BACK02).
When - at what intervals? How long does it take?
Nightly to tape, but T-logs are backing up from SQL every 10-15 minutes. The SQL backups take about 20 minutes for the primary cluster and 40 minutes for the secondary cluster. T-Log backups run for less than a minute on the primary cluster and take about 6 minutes on the secondary (due to the high database count). I think the tapes take a bit at night, mostly limited by disk speed of the DAS.
What level - what all content is actually backed up? How many layers of such backups exist?
Anything important gets backed up. Everything from internal company docs (payroll, etc.) to production SQL. Retention differs by content but generally we don't keep many backups past a few months. SQL is a special case, we keep the first of the month in a separate on-disk archive in both data centers just in case. It's come in handy a few times when tracking down bugs. For the on-disk story we keep SQL fulls back 7 days and T-logs back 3 days.
Also, who has the authority to restore backups (in case one is required)?
Whoever's awake. If you're talking SQL that would be an emergency team call - we'd have to really be hurting for that to happen. Unless you mean test restores, we have a dedicated NY-RESTORESQL01 which does nothing but restore our backups all day to make sure they work.
Has the site been restored before?
Once, when we were in Oregon and lost power hard. Sam Saffron was the main player piecing together the data after an emergency restore. We have become incredibly more resilient to loss since then, with redundant servers, constant backups, multiple data centers lots of automation.
If I missed something let me know - if you want more specifics on the tape details then Pete can answer better when he's back from vacation.
See Failing Over Stack Overflow and Stack Exchange for Sandy and Stack Exchange: Planning for failure, part 1.
Basically, SE has two datacenters: One in NY, one in Oregon. The NY one seems to be the primary one, however they have set it up so that if anything happens to NY (like a hurricane), the system can easily failover to Oregon. This has already been done a couple of times, a few times for testing and once in the face of an actual disaster.
So there is (realtime?) database replication going on here, though there may also be separate backups.
