What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 127 Stack Exchange communities.

I recently saw this image posted in an answer on Meta by @Nick Craver ♦:

a

I'm interested to know how much bandwidth (in total GB/TB) gets used on average each month for such a large site, and on which services? (IIS/SQL split + any other servers)

share|improve this question
add comment

1 Answer

up vote 97 down vote accepted
+50

Some quick stats on the past 30 days, the New York data center only (chat runs from Oregon, and we sync databases there...so there's additional bandwidth and such for that).

HTTP Traffic only:

  • 20,335,940,765,006 bytes sent (18.5 TB)
  • 2,839,549,699 total hits to our load balancers
  • 8,314,044,406 SQL queries run (just for web pages)

Now that's the outward facing traffic, which I assume is what you're after here. Also note that it excludes a significant figure, the content hosted by our CDN. Here's a few figures from our CDN:

  • 15.75TB transferred
  • 1,391,575,359 hits
  • 99.59% cache hit ratio
  • 5,731,000 non-cache hits (hit our network)

Now none of these includes other bandwidth uses, such as the aforementioned database syncs to other data centers and other VPN traffic. We maintain a VPN mesh between all DCs and offices at all times, but SQL replication, redis slaving and offsite backup copies account for the majority of the bandwidth there. For reference, here's total bandwidth data (only the external interface) from the Cisco 3945 on our primary uplink:

  • 6,069,171,969,536 bytes received
  • 22,485,865,626,896 bytes sent (20.4TB)

So you can see there we have about 2.2TB of send overhead just to run things, plus another 6TB just for requests.


For those curious what performance looks like these days, here's our database tier:

database servers

What they do:

  • New York:
    • NY-DB05 - Used for internal monitoring and HAProxy HTTP traffic logging
    • NY-SQL01 - The primary Stack Overflow and global network databases
    • NY-SQL02 - Read-only async replica of NY-SQL01 (queryable, but just a backup currently), also hosts our dev environment on dedicated SSDs
    • NY-SQL03 - Every other site database, Careers, OpenID, Area51, SE.com, promotions, etc.
    • NY-SQL04 - Read-only async replica of NY-SQL03 (again, not used in prod, just a hot backup)
  • Oregon:
    • OR-DESQL01 - data.stackexchange.com and data dumps (one coming very soon)
    • OR-HALOG01 - Equivalent of NY-DB05, for HAProxy HTTP logs
    • OR-SQL01 - Read-only async replica of NY-SQL01, also chat primary server and dev
    • OR-SQL02 - Read-only async replica of NY-SQL03

And here's our web tier:

web servers

What they do:

  • New York:
    • NY-PROMOWEB01-02 - Promotional web sites, such as answerswarm and apptivate
    • NY-WEB01-09 - Q&A (everything but meta and dev), Careers, SE.com, OpenID...everything
    • NY-WEB10-11 - Meta, dev, and some internal sites
  • Oregon:
    • OR-PROMOWEB01-02 - Same as NY, idle but built weekly when not running from Oregon
    • OR-VMWEB01-02 - Equivalent to NY-WEB10-11
    • OR-WEB01-07 - Equivalent to NY-WEB01-09, but with more chat

The graphs are wacky on the last servers because we're taking them out and upgrading the entire tier to Server 2012 as I write this, and we whipped the monitoring back into shape for this post.

One last item for those still paying attention: Most of the bandwidth is still internal. Over the same 30 days our core switches went through 586,157,817,892,376 bytes of traffic.

If there are other things you're curious about, just ask. We're pretty open about how we run things.

share|improve this answer
2  
Are these numbers for Stack Overflow or Exchange? If the former could you add in the numbers for the latter? –  Sathya Mar 18 '13 at 13:16
6  
@Sathya those numbers are for all sites...we'd obviously only have http breakdowns by site, but we do have it. –  Nick Craver Mar 18 '13 at 13:43
    
Are these the custom graphs you made against Orion NPM? –  pauska Mar 18 '13 at 16:21
3  
Only question I have is RE: migration to Server 2012. I run a small operation on a single, moderately beefy enterprise-grade Server 2012 box. Same kinda traffic: lotsa database and web. Things were great with 2008 R2 but we've had nothing but pain with 2012. If you can share your experiences once your migration is complete I'd love to hear about it. Maybe a blog post or something... –  ÃŁŁǫǛȉЖΦΤїҪ Mar 18 '13 at 16:30
    
Great answer, thanks a lot Nick. Very interesting indeed! –  Danny Beckett Mar 18 '13 at 16:31
2  
@pauska - yeah we have an entire dashboard for our internal systems, some of it built on Orion. I really want to open source it but have to find the time to finish the JS charting conversion to d3. –  Nick Craver Mar 18 '13 at 16:51
    
@allquixotic - we didn't have a lot of choice on the 2012 migration on the SQL clustering front, the 2k8r2 clustering being FUBAR really forced our hand. That being said, we haven't had any major issues with 2012...what issues are you seeing? –  Nick Craver Mar 18 '13 at 16:54
1  
@NickCraver mostly with the networking stack's stability and throughput (or lack thereof). It might be a buggy driver in 2012 for our NIC (an Ivy Bridge server chassis Intel NIC) or even possibly faulty hardware, but the DC ops guys say everything from the first hop switch all the way out to the public internet is fully working, and our traffic shape looks normal and not flooded or anything. Yet we experience huge latency spikes, 30 sec and upwards of 100% packet loss every few days, and frequent degradation of throughput. It's really smelling like a driver config or impl bug. –  ÃŁŁǫǛȉЖΦΤїҪ Mar 18 '13 at 17:04
12  
@allquixotic - very odd, and nothing we've seen at all. We're on Intel QP gig nics (embedded dell and standalone) on latest drivers with no issues. We're also using built-in 2012 teaming and LACP bonding across the FEXs and nexus cores. Post a question with the details you have on serverfault.com, we'll take a look as will many network gurus who don't work here :) –  Nick Craver Mar 18 '13 at 17:12
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .