What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 127 Stack Exchange communities.

For a couple of minutes, I got...

Server Error in '/' Application.
Runtime Error
Description: An exception occurred while processing your request. Additionally, 
another exception occurred while executing the custom error page for the first 
exception. The request has been terminated. 

I'm curious what happened... I couldn't even post a question to meta during the error condition...

share|improve this question
13  
At least a cat was working on it right away. The cat's grammar wasn't great but he at least looked competent. –  Jay Riggs Oct 31 '12 at 18:28
3  
I never even saw the cat... I was getting a graphical MS Windows error popup –  Mike Pennington Oct 31 '12 at 18:30
    
@MikePennington I can add a screenshot to your question if you want ;-) –  assylias Oct 31 '12 at 18:34
1  
@JayRiggs: More competent than this dog, anyway. –  Dennis Oct 31 '12 at 18:39
add comment

1 Answer

up vote 11 down vote accepted

We're looking into why you saw an error, but the event was caused by SQL coming online in New York.

We're using SQL 2012 Always On Availability groups as part of our high availability architecture. How this works is the primary database server for that group (these are currently moved to Oregon due to Sandy) streams changes to the replicas.

Due to a lack of power in New York (status updates on our NY datacenter can be found here, here and here), we are keeping our equipment there off as much as we can. However, the replicas for SQL being down mean the backlog of changes needed to sync to them builds up in Oregon, at the rate of about 40GB that needs sending every 6 hours.

To relieve these logs (which we could tolerate for up to ~4 days) we spin up the New York SQL boxes to receive changes and let the logs clear out on the primary. On this latest spin up, the combination of no domain controller in New York online and the SQL boxes coming up simultaneously caused the windows clustering service to fail on NY-SQL01, and while the SQL cluster on top of this windows cluster assessed what was wrong, the databases were unavailable.

The app should have thrown a nice error page during this at least, we'll look into that.

We're by no means sitting around idle after our failover to Oregon. There are many, many things still happening, but we'll try and keep the interruptions to a minimum.

share|improve this answer
6  
Love the transparency when it comes to these things. –  Mysticial Oct 31 '12 at 19:40
1  
Why are both datacenters in coastal areas? It would be nice if there was something further inland. –  NullUserException อ_อ Oct 31 '12 at 19:45
    
@NullUserExceptionอ_อ Right into tornado alley! –  asawyer Oct 31 '12 at 19:45
4  
@NullUserExceptionอ_อ the unicorns that power the system insist on somewhere near the sea - they like boats. –  Marc Gravell Oct 31 '12 at 19:46
1  
Lack of transparency typically starts with abundance of unwarranted criticism, or an abundance of technical buffoonery. –  Mike Pennington Oct 31 '12 at 19:47
    
How far down the “5 Whys” do you get before global warming and the size of cars in the US? –  Ian Ringrose Oct 31 '12 at 22:51
1  
Keep Oregon as your primary - it's closer to where I live so the bits get to me faster. –  slugster Oct 31 '12 at 22:57
    
@NullUserExceptionอ_อ: Corvallis is well protected by the Oregon Coast Mountain Range from the Pacific Ocean which is 40 miles away—unlike NYC which is basically at sea level with no protection. There is a theoretical possibility of Corvallis flooding from the Willamette River, but dozens of large flood control dams built in the Cascades have, so far, adequately controlled that peril. The only other dangers are earthquakes, which we are pretty well prepared for, and volcanoes, which are of more danger to Portland 70 miles to the north. –  wallyk Nov 1 '12 at 18:24
    
I am getting this error since a month. Not been able to resolve. I tried multiple browsers thinking could be a problem in browser but all fail with same error. Interestingly this happens only on my system. No such error on other systems –  Shashi Feb 26 '13 at 11:39
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .