10

With some gaps, I am today getting a lot of 523 errors like those shown here:

The CloudFlare location might say Ashburn, or Newark, etc.; it doesn't seem to matter. The "additional troubleshooting information" link points here, recommending some changes to the CloudFlare DNS Settings page.

I also have increased CAPTCHAs (having to click "I'm Not a Robot" a lot on this site today, which then don't register when processed by 523 errors), and then sometimes see the Oops! Something Bad Happened error. Additional feedback is provided here as requested.

Is SE under a DDoS attack?

Any idea what the cause is, how to interpret the error, or how to fix it?

11

We ran into issues upgrading our edge routers that talk to cloudflare.

TL;DR; We ran into a LACP bug so depending on how the traffic was hashed once it hit our infrastructure it may or may not have traveled down a LAG member port that worked.

The Long version:

We run Cisco Nexus 5k's in a redundant pair using Virtual Port Channels. When installing the first router for the upgrade, I had not setup the VPC on both sides. This caused the LACP bond on the router to flap and break in new and interesting ways. Even though the router at that time was only plugged into one of the Nexus Cores.

The expected behavior was it would bring up the LACP bond with only one member. This did not happen. Once we got the second VPC configured on the second Nexus, the interfaces and bonds came up clean. However LACP was in a strange state. It connectivity issues. Once we found the problem we verified all services where on the known good router as expected and rebooted the new router to allow everything to come up fresh.

After the reboot we verified that connectivity across the bond was good.

We have left the system in a known good state for the night. One new router and one old router. This upgrade was for future projects so capacity is not an issue right now. I will continue the hardware upgrade tomorrow after everyone has had time to rest and look at things fresh in the morning.

  • 1
    +1 it's great to see a company honest and open enough with its users to take the time to explain something like this. * hat-tip * – user56reinstatemonica8 Jul 1 '15 at 9:48
  • "break in new and interesting ways" Gotta love modern technology - always keeping even experienced techies on their toes :D – James Jul 1 '15 at 14:19
12

https://twitter.com/StackStatus/status/616013999969140736

  • Did @nicael break something? – user289879 Jun 30 '15 at 23:21
  • 4
    @santiago He's good, but not that good. (Yet? :)) – Adam Lear Jun 30 '15 at 23:21
  • @AnnaLear hush, he may take that as a challenge...lol – user289879 Jun 30 '15 at 23:22
  • 6
    Almost certainly yes, @santiago. But unrelated to this. – Shog9 Jun 30 '15 at 23:24
  • 6
    I have broken the cover of my galaxy s5!!! Now it says itsnt attached properly!!! – nicael Jun 30 '15 at 23:25
  • OMG That's the first step! – user289879 Jun 30 '15 at 23:27

Not the answer you're looking for? Browse other questions tagged .