15

https://stackstatus.tumblr.com/post/37789095378/outage-dec-12th-2012

Even though the primary load balancer appears to be fine after the kick, we’re not trusting it just yet and have shifted it back in rotation while we fully investigate.

What was the result of this investigation about the primary load balancer?

Improve this question

1 Answer 1

Reset to default
12

In our Oregon datacenter, our load balancers were also serving as the VPN endpoints for our site-to-site network. The IPSec stack has some oddities in it that would cause certain kernel buckets to be overwhelmed when there is high traffic activity (See this blog post for some more background.) We found some ways to improve the situation, but we ultimately realized we needed to move the VPN off of the load balancers.

We were intending to put Cisco ASA's in place anyway, so we accelerated that project and swapped the VPN over. The problem hasn't recurred since.

Improve this answer
3
  • Nice, can it be added to the outage post, or are those only for instant status report?
    – Shadow Wizard
    Commented Apr 18, 2013 at 15:00
  • We generally use stackstatus.net just to inform users of immediate problems. It might be an interesting topic for blog.serverfault.com, but since the problem is no longer relevant to our experience it'd be tough to get specific information about it to have a truly well-informed post about it.
    – Peter Grace Mod
    Commented Apr 18, 2013 at 15:02
  • Fair enough, we have this post now for reference if needed. :)
    – Shadow Wizard
    Commented Apr 18, 2013 at 15:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .