The data is updated early every Sunday morning around 3:00 UTC
Update 9 Nov 2012 - Done!
Since our datacenter move has been delayed, I was able to use the new server as soon as it arrived. Data.SE now has fresh data and the import is almost fully automated. I'll finish the automation on Monday and we'll have a weekly refresh of data going on. On Monday we'll also get the graduated sites that are missing in there - we haven't forgotten about them.
There may be issues with this new build that we're unaware of - it's a whole new setup. Please comment here if you notice any issues.
Update: we're just waiting on the new server to arrive (ETA 2012-11-09) in Oregon so I can get Data.SE on an auto-update. We should be able to get it up about a week after it comes in (it arrives the same weekend as our New York datacenter move, so it will likely have to sit a few days).
To clarify/set expectations: for data.SE we're talking about a weekly refresh (that's my goal, since it has to be down for a moment when we reload fresh data and it takes time to run). The SEDE George (Zypher) was talking about in chat was actually our internal copy of SEDE (which is just raw data, not transformed or anonymized - much easier to do "live").
We call that "LocalSEDE" internally and just "SEDE" for the public data.SE. So while it will be much fresher, it won't be instant. To get data there we have to transform and move it, and we can't logistically do that continuously, at least not in the foreseeable future.
Rebecca is the one taking lead on this as I'm working on the infrastructure side, but we'll update here in case we run into anything that makes weekly an issue.
Also to clarify, for data.SE we're talking about a weekly refresh or so (that's my goal, since it has to be down for a moment when we reload fresh data).
We have uncovered a critical bug in our network setup paired with SQL Server 2012 clustering... We'll get to the data refresh when Stack Overflow and chat are no longer in danger of going offline. We're working with Microsoft on the issue now.
Data Explorer lives in Oregon, a faster data refresh is predicated on getting a replica up and running out there for all databases, which currently exists for Stack Overflow but not any other sites. To get a data refresh going we need to:
- Get the bug fixed and deploy that fix across our network
- Get all other sites moved to the second SQL Server 2012 cluster
- Move chat to SQL Server 2012
- Re-task the chat database server as a 2012 instance that SEDE will run on
- Move SEDE to the new server
- Re-write the data import process for SEDE to be totally automated and pull from the replicas
We plan on doing all of this, but our primary concern is getting our architecture fixed and moved to support this (for other, more important reasons like moving data centers really). Once that's done then we'll get to SEDE data refresh. Since we are shuffling hardware here and SEDE is dependent on data, it would be a large amount of throw-away effort to get anything working before the above is complete.
The result of this will be no more manual imports, but (hopefully) something like a weekly data refresh to SEDE rather than the somewhat random intervals it's updated on currently. I apologize that things aren't going as planned with our timeline, trust me when I say we are far more frustrated with this than you are. Bear with us, the result will be worth it. SEDE's getting fresher, more reliable data refreshes and a beefier database server to run on.