We ran into issues upgrading our edge routers that talk to cloudflare.
TL;DR; We ran into a LACP bug so depending on how the traffic was hashed once it hit our infrastructure it may or may not have traveled down a LAG member port that worked.
The Long version:
We run Cisco Nexus 5k's in a redundant pair using Virtual Port Channels. When installing the first router for the upgrade, I had not setup the VPC on both sides. This caused the LACP bond on the router to flap and break in new and interesting ways. Even though the router at that time was only plugged into one of the Nexus Cores.
The expected behavior was it would bring up the LACP bond with only one member. This did not happen. Once we got the second VPC configured on the second Nexus, the interfaces and bonds came up clean. However LACP was in a strange state. It connectivity issues. Once we found the problem we verified all services where on the known good router as expected and rebooted the new router to allow everything to come up fresh.
After the reboot we verified that connectivity across the bond was good.
We have left the system in a known good state for the night. One new router and one old router. This upgrade was for future projects so capacity is not an issue right now. I will continue the hardware upgrade tomorrow after everyone has had time to rest and look at things fresh in the morning.