I just logged in to The Comms Room (ServerFault chat) and noticed this:

24 days?

Based on comments and quick look at a few rooms, seems to be all sites except Tavern on the meta.

Something seems amiss...

  • I'm seeing this as well in the R chatroom. We seem to have travelled back in time 24 days...? – joran Oct 9 '12 at 14:26
  • I'm seeing this across the board in all the chatrooms across the entire chat network. – jrg Oct 9 '12 at 14:27
  • Black hole. Also we got hardcore logged out and I had to log in again via stack exchange – Ben Brocka Oct 9 '12 at 14:34
  • Looks like it's back to normal...? – joran Oct 9 '12 at 14:36
  • The Comms Room is back... – Ward Oct 9 '12 at 14:37
  • I noticed this. The big gap is no longer missing, but now there are a few more recent messages missing. – Kendall Frey Oct 9 '12 at 14:45
  • 6
    Gremlins. These are not the chat logs you're looking for. We have always been at war with Stack Overflow... – voretaq7 Oct 9 '12 at 14:50
  • 3
    "There's a chance of one in five that you experienced some time travel in the past half hour. This should be fixed now, but a few messages may have been lost (but most people wouldn't have seen your message anyway). Sorry about that." You'd better be sorry. I want my time travel back. – R. Martinho Fernandes Oct 9 '12 at 14:58

Sorry, this was my fault. We started about 2 weeks ago building our entire external infrastructure out in our Oregon DC (Stack Overflow's original home). To do this I copied the sites on our new OR-WEB01 -> OR-WEB04 servers over to the even newer OR-WEB05 to do this build out of the line of fire.

Well, that plan only lasts so long, eventually I need to deploy that template OR-WEB05 server to the rest of the tier...which just happened about an hour ago. Along with that we added OR-WEB05 to our haproxy load balancers out there on all the backends that matter (including chat).

So what went wrong? Chat hasn't been building on this new server (so I wouldn't break builds), so it has old connection strings pointing to a SQL 2008 server before our SQL 2012 move. We were going to nuke that box which would have prevented this from happening (by throwing errors and HAProxy taking it out), but that didn't happen for other reasons.

We made a few changes to further ensure this doesn't happen again (the SQL 2008 databases are gone this time), and we should be set from now on.

We're making a lot of changes to our Oregon datacenter where chat runs. We're doing our best to ensure you don't even know about it until we actually turn the lights on, so to speak.

This Saturday we'll be doing a test run of hosting the entire Q&A network in Oregon (read/write, not read-only), while we're testing everything we can ahead of time, there may be issues...that's why it's a test. We'll be doing another test in a few weeks with Careers, openid.SE, and Apptivate. We'll get a blog post up on what the plans are today or tomorrow at the latest for those that are curious, and a heads up to everyone.

  • 2
    Great one for the coffee table, and thanks for the details. – Toon Krijthe Oct 9 '12 at 15:11
  • 2
    And don't forget updating data.SE.. – Ward Oct 9 '12 at 16:23
  • 3
    A picture of @Nick sorting this out can be found here – Tim Post Oct 9 '12 at 23:37
  • 1
    You guys rock. Never forget transparency rules the roost. – Max Vernon Oct 11 '12 at 4:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .