What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 133 Stack Exchange communities.

As a result of power loss at the PEAK data center as posted on our blog our SQL instance was shut down unexpectedly. SQL Server was unable to recover using dbcc checkdb and we decided to restore from a backup taken at 10/9/2010 00:00 PST. We do have the data and will work quickly to restore as much as we can from the timeframe between the last backup and 4:00 PST when the outage occurred. We will update this post as we have more information.

update

We have finished recovering the missing posts and post history. It was a manual process. We are still working on recovering some of the votes and comments.

update 2

comments are recovered

update 3

I am done (waffles out) ... finished migrating votes as well

share|improve this question

closed as too localized by Rory, Pops, Lance Roberts, Toon Krijthe, Bo Persson Dec 11 '12 at 16:34

This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

    
Um, 8/12/2010 is a typo I assume? –  Kirk Woll Oct 9 '10 at 16:18
    
Yes, it's wayy to early for this. –  Geoff Dalgas Oct 9 '10 at 16:21
2  
Ah, the wonders of SQL Server. –  Rosinante Oct 9 '10 at 22:27

5 Answers 5

Nice job!

One thing to note: post IDs and user IDs might have changed, probably implying that search engine results may be off for a while too.

(Update, 2010-10-11: searching Google for the titles of the examples below, now gets one the correct posts. But other searches might still yield wrong results.)

Like:

Also: some post may show "show xx more comments" while clicking doesn't reveal them. This will probably be solved automatically by the daily process recalculating the denormalized count.

share|improve this answer
2  
Not so tough now, eh, Mr. Exploit-the-Hamsters-to-the-max?! –  Mark C Oct 9 '10 at 16:44

Some restored posts have wrong vote decomposition, e.g. in http://stackoverflow.com/questions/3896802/c-programming-language-increment-decrement-operators,

alt text

share|improve this answer
    
The total count is denormalized too. Will probably be fixed when the daily scripts run? –  Arjan Oct 9 '10 at 16:56
    
votes are not recovered yet, its my last task –  waffles Oct 9 '10 at 17:03
3  
just fixed this ... I'm done for tonight –  waffles Oct 9 '10 at 17:14
    
@Arjan: That answer is upvoted 3 times, so the correct decomposition should be +3/-0, i.e. the denormalized vote is wrong. –  KennyTM Oct 9 '10 at 17:14
    
@waffles: Thanks. –  KennyTM Oct 9 '10 at 17:17
    
@waffles: Are you guys done with the recovery efforts? This answer has the same issue as the one Kenny mentioned. Received one vote during the downtime. Also an answer I added is gone (including the question too). I imagine there are other smaller discrepancies in others as well. –  Jeff Mercado Oct 9 '10 at 18:04
    
I believe I'm still observing this problem and am missing the 20 rep from the two "ghost" downvotes on stackoverflow.com/questions/3895525/… despite my history showing 8 upvotes from that same answer... –  Mark Elliot Oct 9 '10 at 22:17
    
Nice red ... chunky –  Aiden Bell Oct 9 '10 at 23:15
    
Still problematic at this moment for a few posts... –  ChristopheD Oct 9 '10 at 23:18
    
@waffles Is rep of users supposed to take votes into account? Mine doesn't. Is that normal? Should I report something? –  Pascal Thivent Oct 9 '10 at 23:50
    
lets wait a day for the daily jobs to run, then we can do a rep recalc for any users noticing weirdness –  waffles Oct 10 '10 at 1:08
    
in a day or so add questions for any weird leftovers so I can clean up manually –  waffles Oct 10 '10 at 1:09
    
I (user gorohoroh) have voted for answer stackoverflow.com/questions/3355289/… and seems like the vote didn't count. Same for me: I received a couple of votes for my answers (for example, this one: stackoverflow.com/questions/3707538/…) on the weekend, and my points for the votes were rolled back although the vote count in my profile does reflect the votes. Please check this. Thanks –  gorohoroh Oct 11 '10 at 19:26

Looks like some votes are still missing. I lost 35 points.

Also I had upvoted 3 answers on this question http://stackoverflow.com/questions/3896749/which-encryption-method-would-produce-this-result. I have re-upvoted them after the crash.

This question http://stackoverflow.com/questions/3896564/large-table-querying/3896583#3896583 is showing up as accepted and upvoted on my recent activity, but not on the question.

This question http://stackoverflow.com/questions/3892527/wcf-how-to-diagnose-faulted-channels/3893164#3893164 also shows up as an upvote on my recent activity, but has 0 upvotes on the question.

These 2 questions match the missing 35 points.

share|improve this answer

The metadata for question http://stackoverflow.com/questions/3883692/strategy-pattern-vs-command-pattern is incorrect. I noticed answer count and latest activity are under-represented. (I added an answer during the period lost due to the restore.) I don't know if any others are also incorrect.

share|improve this answer

I'm just asking -- I've drunk the kool-aid that makes me believe that when a SQL Server database is running using the "full recovery model" (with log backups being taken regularly) that the only possible data loss that can occur (other than the situation where the disk holding the current log backup is physically damaged by the power outage) is that any transactions in progress at the time of the outage will be rolled back (but all that were successfully committed will be recovered).

Unless there is physical disk damage, when power is restored, isn't SQL Server supposed to be able to recover automatically? You say that it marked the database as "suspect". What did the SQL Server event log say was the cause of it being so marked? Did you make a copy of the database and log files as they were at that point and open a case with Microsoft to let them figure out why the recovery didn't happen? SQL Server -- I thought -- is very careful about the sequence in which it writes to the database file(s) and log file so that recovery is guaranteed.

Am I wrong that SQL Server is designed to handle exactly this situation 100.00% of the time??? Isn't it a bug -- or a hardware failure -- when it doesn't?

Even if SQL Server can't process the database and log files when it restarts (possibly due to disk damage), if you obtain functioning disk hardware, restore the most recent backup, and apply all subsequent log backups followed by the current live log file (if available), shouldn't that bring you to exactly the point of the last successfully committed transaction? If the current live log file isn't available, then you "only" get to the point of the most recent transaction log backup -- which in a busy site I would think would be maybe 10 minutes old, not hours.

I'm concerned that what happened to SO's database could happen to one of mine, so I'm wanting to know more than I do now about what exactly was wrong and whether MS has had a chance to see the problem "live" and as a result possibly make (some version of) SQL Server more robust in that situation.

share|improve this answer
    
we did not have a log backup at the time, and the mdf was suspect. if we were backing up logs we could have just replayed back to the last log backup. these days we are backing up every 5 minutes, back then we did full backups daily and diffs every 6 hours. .... sql is supposed to handle power outages just fine, but there are complicating factors, in this case it did not. MS did not look at our corrupt MDF yet, not sure if we still have it around. –  waffles May 17 '11 at 2:08
    
@waffles -- Thanks for the explanation; without log backups you definitely have fewer recovery options. What did the event log say was the cause of the MDF being marked "suspect"? –  J.Merrill May 18 '11 at 20:15

Not the answer you're looking for? Browse other questions tagged .