14

For a couple of minutes, I got...

Server Error in '/' Application.
Runtime Error
Description: An exception occurred while processing your request. Additionally, 
another exception occurred while executing the custom error page for the first 
exception. The request has been terminated.

I'm curious what happened... I couldn't even post a question to meta during the error condition...

Improve this question
4
  • 13
    At least a cat was working on it right away. The cat's grammar wasn't great but he at least looked competent.
    – Jay Riggs
    Oct 31, 2012 at 18:28
  • 3
    I never even saw the cat... I was getting a graphical MS Windows error popup
    – Mike Pennington
    Oct 31, 2012 at 18:30
  • @MikePennington I can add a screenshot to your question if you want ;-)
    – assylias
    Oct 31, 2012 at 18:34
  • 1
    @JayRiggs: More competent than this dog, anyway.
    – Dennis
    Oct 31, 2012 at 18:39

1 Answer 1

Reset to default
12

We're looking into why you saw an error, but the event was caused by SQL coming online in New York.

We're using SQL 2012 Always On Availability groups as part of our high availability architecture. How this works is the primary database server for that group (these are currently moved to Oregon due to Sandy) streams changes to the replicas.

Due to a lack of power in New York (status updates on our NY datacenter can be found here, here and here), we are keeping our equipment there off as much as we can. However, the replicas for SQL being down mean the backlog of changes needed to sync to them builds up in Oregon, at the rate of about 40GB that needs sending every 6 hours.

To relieve these logs (which we could tolerate for up to ~4 days) we spin up the New York SQL boxes to receive changes and let the logs clear out on the primary. On this latest spin up, the combination of no domain controller in New York online and the SQL boxes coming up simultaneously caused the windows clustering service to fail on NY-SQL01, and while the SQL cluster on top of this windows cluster assessed what was wrong, the databases were unavailable.

The app should have thrown a nice error page during this at least, we'll look into that.

We're by no means sitting around idle after our failover to Oregon. There are many, many things still happening, but we'll try and keep the interruptions to a minimum.

Improve this answer
9
  • 6
    Love the transparency when it comes to these things.
    – Mysticial
    Oct 31, 2012 at 19:40
  • 1
    Why are both datacenters in coastal areas? It would be nice if there was something further inland.
    – NullUserException อ_อ
    Oct 31, 2012 at 19:45
  • @NullUserExceptionอ_อ Right into tornado alley!
    – asawyer
    Oct 31, 2012 at 19:45
  • 4
    @NullUserExceptionอ_อ the unicorns that power the system insist on somewhere near the sea - they like boats.
    – Marc Gravell
    Oct 31, 2012 at 19:46
  • 1
    Lack of transparency typically starts with abundance of unwarranted criticism, or an abundance of technical buffoonery.
    – Mike Pennington
    Oct 31, 2012 at 19:47
  • How far down the “5 Whys” do you get before global warming and the size of cars in the US?
    – Ian Ringrose
    Oct 31, 2012 at 22:51
  • 1
    Keep Oregon as your primary - it's closer to where I live so the bits get to me faster.
    – slugster
    Oct 31, 2012 at 22:57
  • @NullUserExceptionอ_อ: Corvallis is well protected by the Oregon Coast Mountain Range from the Pacific Ocean which is 40 miles away—unlike NYC which is basically at sea level with no protection. There is a theoretical possibility of Corvallis flooding from the Willamette River, but dozens of large flood control dams built in the Cascades have, so far, adequately controlled that peril. The only other dangers are earthquakes, which we are pretty well prepared for, and volcanoes, which are of more danger to Portland 70 miles to the north.
    – wallyk
    Nov 1, 2012 at 18:24
  • I am getting this error since a month. Not been able to resolve. I tried multiple browsers thinking could be a problem in browser but all fail with same error. Interestingly this happens only on my system. No such error on other systems
    – Shashi
    Feb 26, 2013 at 11:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .