How much bandwidth does SE eat up?

Question

I recently saw this image posted in an answer on Meta by @Nick Craver ♦:

I'm interested to know how much bandwidth (in total GB/TB) gets used on average each month for such a large site, and on which services? (IIS/SQL split + any other servers)

Answer 1

Some quick stats on the past 30 days, the New York data center only (chat runs from Oregon, and we sync databases there...so there's additional bandwidth and such for that).

HTTP Traffic only:

• 20,335,940,765,006 bytes sent (18.5 TB)
• 2,839,549,699 total hits to our load balancers
• 8,314,044,406 SQL queries run (just for web pages)

Now that's the outward facing traffic, which I assume is what you're after here. Also note that it excludes a significant figure, the content hosted by our CDN. Here's a few figures from our CDN:

• 15.75TB transferred
• 1,391,575,359 hits
• 99.59% cache hit ratio
• 5,731,000 non-cache hits (hit our network)

Now none of these includes other bandwidth uses, such as the aforementioned database syncs to other data centers and other VPN traffic. We maintain a VPN mesh between all DCs and offices at all times, but SQL replication, redis slaving and offsite backup copies account for the majority of the bandwidth there. For reference, here's total bandwidth data (only the external interface) from the Cisco 3945 on our primary uplink:

• 6,069,171,969,536 bytes received
• 22,485,865,626,896 bytes sent (20.4TB)

So you can see there we have about 2.2TB of send overhead just to run things, plus another 6TB just for requests.

---

For those curious what performance looks like these days, here's our database tier:

What they do:

• New York:
  • NY-DB05 - Used for internal monitoring and HAProxy HTTP traffic logging
  • NY-SQL01 - The primary Stack Overflow and global network databases
  • NY-SQL02 - Read-only async replica of NY-SQL01 (queryable, but just a backup currently), also hosts our dev environment on dedicated SSDs
  • NY-SQL03 - Every other site database, Careers, OpenID, Area51, SE.com, promotions, etc.
  • NY-SQL04 - Read-only async replica of NY-SQL03 (again, not used in prod, just a hot backup)
• Oregon:
  • OR-DESQL01 - data.stackexchange.com and data dumps (one coming very soon)
  • OR-HALOG01 - Equivalent of NY-DB05, for HAProxy HTTP logs
  • OR-SQL01 - Read-only async replica of NY-SQL01, also chat primary server and dev
  • OR-SQL02 - Read-only async replica of NY-SQL03

And here's our web tier:

What they do:

• New York:
  • NY-PROMOWEB01-02 - Promotional web sites, such as answerswarm and apptivate
  • NY-WEB01-09 - Q&A (everything but meta and dev), Careers, SE.com, OpenID...everything
  • NY-WEB10-11 - Meta, dev, and some internal sites
• Oregon:
  • OR-PROMOWEB01-02 - Same as NY, idle but built weekly when not running from Oregon
  • OR-VMWEB01-02 - Equivalent to NY-WEB10-11
  • OR-WEB01-07 - Equivalent to NY-WEB01-09, but with more chat

The graphs are wacky on the last servers because we're taking them out and upgrading the entire tier to Server 2012 as I write this, and we whipped the monitoring back into shape for this post.

One last item for those still paying attention: Most of the bandwidth is still internal. Over the same 30 days our core switches went through 586,157,817,892,376 bytes of traffic.

If there are other things you're curious about, just ask. We're pretty open about how we run things.