What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 134 Stack Exchange communities.

http://stackstatus.net/post/37789095378/outage-dec-12th-2012

Even though the primary load balancer appears to be fine after the kick, we’re not trusting it just yet and have shifted it back in rotation while we fully investigate.

What was the result of this investigation about the primary load balancer?

share|improve this question

1 Answer 1

up vote 13 down vote accepted

In our Oregon datacenter, our load balancers were also serving as the VPN endpoints for our site-to-site network. The IPSec stack has some oddities in it that would cause certain kernel buckets to be overwhelmed when there is high traffic activity (See this blog post for some more background.) We found some ways to improve the situation, but we ultimately realized we needed to move the VPN off of the load balancers.

We were intending to put Cisco ASA's in place anyway, so we accelerated that project and swapped the VPN over. The problem hasn't recurred since.

share|improve this answer
    
Nice, can it be added to the outage post, or are those only for instant status report? –  Shadow Wizard Apr 18 '13 at 15:00
    
We generally use stackstatus.net just to inform users of immediate problems. It might be an interesting topic for blog.serverfault.com, but since the problem is no longer relevant to our experience it'd be tough to get specific information about it to have a truly well-informed post about it. –  Peter Grace Apr 18 '13 at 15:02
    
Fair enough, we have this post now for reference if needed. :) –  Shadow Wizard Apr 18 '13 at 15:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .