I think there are multiple people aware of this, but I want to start a question for it to open a discussion and for admins to post relevant details.

I've noticed (and from reading comments here and elsewhere, others have too) that there is an intermittent failure which results in a database error page, or the standard StackOverflow error page. I'll try to screen cap it next time it happens.

I'd assume admins likely already know about this from error logs and server messages. Just wondering what's going on here.

EDIT: So, looking at both the screenshots, it looks like there's really only one error in the back-end. The other screenshot just says that it failed to properly display the error message.

  • I have a screen capture of both error pages if you want me to post them. – user208121 Jan 14 '13 at 2:26
  • Sure! Comment them here and I'll add them to the post. – user206222 Jan 14 '13 at 2:27
  • Dont bother except for Jon Skeet, we've all seen a YSOD... – Jeremy Thompson Jan 14 '13 at 3:11
  • 3
    They should post a question on SO to see if anyone knows a solution ;-) – PeterJ Jan 14 '13 at 8:15

Windows 2008 R2 server clustering on top of which SQL 2012 Always on Availability groups are built is buggy as hell. We're tracking down the YSOD cause now and attempting to fix before doing the next patch round.

To add to the fun, we've had memory and/or a processor memory controller fail in a server in New York. Apologies for the errors, but expect more fun...that's why we're doing this on a Sunday night.

Update: it wasn't the processor, which we replaced at 3 am, dammit.

Annnnnnd, after powering down New York, (but this time on a 15 minute delay, so we were well away from the data center, touché windows) clustering went offline again. But hey! at least we found out a) our NY office building is locked tight after 3am, and b) our mobile hotspots work.

| improve this answer | |
  • Just curious. Is all the traffic pouring in from Reddit and HN right now complicating this? Or is that still just a tiny portion of the entire site's traffic for this time of the day/week? – Mysticial Jan 14 '13 at 2:41
  • 1
    @Mysticial - nah we haven't even noticed that traffic...we're doing some patching and shutdown before shutting NY down for the move. We've had issues with the clustering before, so we opted for a Sunday night operation. – Nick Craver Jan 14 '13 at 2:48
  • 3
    "we haven't even noticed that traffic" - wow, says something about how much traffic the network gets... – Mysticial Jan 14 '13 at 3:00
  • Good luck, thx for the update, and thanks for staying bleeding edge with a great site! – Jeroen Jan 14 '13 at 8:23
  • Does this impact OCD attempts to get the 100-day every-day login badge? I wanna get that badge... – Lizz Jan 14 '13 at 8:38
  • Sunday? But it's Monday morning in Finland (Europe), workday started hours ago. Yes, I surf SO to get my work done :) – JOM Jan 14 '13 at 8:43
  • @Mysticial According to stackexchange.com/sites#traffic, SO gets 4.4M visits per day. Assuming all visits (43k) to your question came in one day, that would still only be around 1% of the traffic SO gets on average – Yi Jiang Jan 14 '13 at 12:02

Here are the two related error pages I took a screen shot of when the site went down in the past hour.

enter image description here enter image description here

| improve this answer | |
  • Thanks but it only helps if CustomErrors was off or if we were viewing the site from the server - otherwise we cant see the actual error message. – Jeremy Thompson Jan 14 '13 at 3:08
  • @JeremyThompson you sort of can, view source for the error page & you'll see the actual error as a HTML comment – Sathyajith Bhat Jan 14 '13 at 4:14
  • 2
    @Sathya That's not particularly doable from a screenshot – Michael Mrozek Jan 14 '13 at 8:27

Here's another one giving me an error:

immediately redirects to the error page

which displays

Server Error in '/' Application.
Runtime Error
Description: An exception occurred while processing your request. Additionally, another exception occurred while executing the custom error page for the first exception. The request has been terminated.

when logged in. In a different browser session (logged out), it seems to hang.

It seems chat is also down: http://chat.stackexchange.com/error?aspxerrorpath=/

Back up as of 2013-01-14T08:21Z.

| improve this answer | |
  • See the updated answer above for the outage you observed, still - reports appreciated! – Nick Craver Jan 14 '13 at 14:35