33

Cannot access SEDE at all.

Have checked with https://downforeveryoneorjustme.com/data.stackexchange.com which also says it's down.

Improve this question
2
  • 13
    Yes, we know and are working on it. We had an issue with the core switches in Colorado last night that put all of our servers in that site in a really bad state. We'll be working on it today to try and get it resolved.
    – Taryn
    Feb 27, 2020 at 12:07
  • 11
    RIP SEDE :( (raises moderator flag at half mast)
    – Robert Columbia
    Feb 27, 2020 at 14:57

1 Answer 1

Reset to default
35

Update: 2020-02-29

We are still dealing with hardware issues in Colorado and are hoping to have them resolved early next week.

In the meantime, we were able to temporarily create a version of Data Explorer that is running out of our NY data center. I setup up a new SQL instance and ran the refresh process yesterday afternoon. It should be available now for use. Once we get Colorado back in a working state, we'll most likely point things back there.

But go forth and query away.

We're aware of the issue and are actively working on resolving it.

For some context about what's going on.

On 2020-02-26 we initialized maintenance on the core switches in our Colorado data center. The purpose of the maintenance was to push some software patches to the switches and should have been routine with a minor blip to the network out there. It turned out not that easy. Basically the software patches have resulted network instability for all the servers in CO.

Unfortunately, SEDE runs solely out of the Colorado data center, and we haven't been able to stabilize the connection for the application. We have a ticket opened with Cisco to assist with the issue and are working to fix it as soon as possible.

I'll update the answer once we have more details and a possible timeline for a resolution.

We're sorry for the extended outage on it.

Improve this answer
5
  • 11
    I broke it :( The intent of this maintenance was to update to our core switches to permanently fix CVE-2020-3120, one of the "CDPwn" vulnerabilities - we disabled CDP on these systems to mitigate any potential exposure when it was announced, but it was time to apply security patches for this and other vulnerabilities. Major packet loss for a reason we can't determine started occurring after the second member of the pair had finished the upgrade, and our attempt to downgrade and roll back failed.
    – Shane Madden
    Feb 27, 2020 at 17:57
  • 1
    @ShaneMadden wouldn't it be possible to temporarily change the IP address to the NY dc so you can at least run a "Down for maintenance" page on it instead of the no connection at all? Assuming you have no ETA from Cisco.
    – rene
    Feb 27, 2020 at 18:03
  • 5
    @rene We actually don't have the app setup on any of the NY web servers, so it's not as easy as pointing the IP over there. SEDE runs everything out of CO. :(
    – Taryn
    Feb 27, 2020 at 18:21
  • 2
    @Taryn Is there any plans to either make it possible to run SEDE out of the NY servers, or at least have a working fallback on the NY servers with a meaningful error message?
    – Laf
    Feb 27, 2020 at 18:56
  • 7
    @Laf Yes, I would definitely like to do that and it's been on my list for a while. It comes down to having hardware, etc to put the SQL instance on. I don't think it will happen in the next few days though.
    – Taryn
    Feb 27, 2020 at 18:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .