15

I went to check some of my queries against UNIX & Linux this morning and noticed that the results all had dates older than a week. Going to rene's "did the SEDE refresh work?" query, I see that U&L is among the sites whose most recent posts/comments are 2020-04-19 instead of 2020-04-26, as expected. Partial results of incompletely-refreshed sites are:

...
StackExchange.Meta                    2020-04-19 05:15:09 comments 
StackExchange.Ubuntu                  2020-04-19 05:08:40 comments 
StackExchange.Stats                   2020-04-19 05:06:54 comments 
StackExchange.Ubuntu                  2020-04-19 05:02:17 posts    
StackExchange.Programmers             2020-04-19 04:51:22 comments 
StackExchange.Programmers             2020-04-19 04:48:14 posts    
StackExchange.Gaming                  2020-04-19 04:46:57 posts    
StackExchange.Stats                   2020-04-19 04:40:08 posts    
StackExchange.Meta                    2020-04-19 04:36:48 posts    
StackExchange.English                 2020-04-19 04:34:46 comments 
StackExchange.Math                    2020-04-19 04:32:38 comments 
StackExchange.Tex                     2020-04-19 04:25:59 posts    
StackExchange.Math                    2020-04-19 04:22:14 posts    
StackExchange.Tex                     2020-04-19 04:19:32 comments 
StackExchange.Unix                    2020-04-19 04:17:12 comments 
StackExchange.English                 2020-04-19 04:15:42 posts    
StackExchange.Unix                    2020-04-19 04:15:31 posts    
StackExchange.Gis                     2020-04-19 04:14:44 posts    
StackExchange.Electronics             2020-04-19 04:11:13 comments 
...

Sorting by date doesn't make it obvious to me where the refresh died/stopped, but hopefully this post alerts the correct people.

Improve this question
5
  • 7
    It seems something failed: twitter.com/tarynpivots/status/1254423433686867969?s=20, apparently because of the new close reasons: twitter.com/tarynpivots/status/1254435296906641409?s=20
    – Tinkeringbell Mod
    Commented Apr 26, 2020 at 16:13
  • 8
    Thanks, @Tinkeringbell! To taryn's "do I fix it today?" -- please do not work if you're not scheduled to, Taryn! SEDE can wait :)
    – Jeff Schaller
    Commented Apr 26, 2020 at 16:15
  • 7
    Yes I’m aware of this. I know what caused it but we need to discuss possible fixes for it. I unbroke it enough to allow for querying across all sites. We’ll look more tomorrow and I’ll kick the job once things have been fixed.
    – Taryn StaffMod
    Commented Apr 27, 2020 at 0:38
  • @Taryn I recall a discussion, possibly with Catija, appropriately a month ago, where there was some delay/breakage in SEDE's updating - might be hidden in comments on MSE, probably not in The Tavern (but, maybe).
    – Rob
    Commented Apr 27, 2020 at 2:47
  • I have created this SEDE Query that gives insight in what the current state is for all SEDE databases and if there is a restore going on.
    – rene Mod
    Commented May 18, 2020 at 19:58

1 Answer 1

Reset to default
11

This should be resolved soon. I have kicked off the refresh process and it's made it past the previous error that resulted in the failure. The full refresh takes about 6-7 hours, so it will be a little while for all the databases to refresh.

Here is a quick summary of the issue.

We recently made a change to the Close UI, and in making those changes, we updated the interface for moderators to manage specific close reasons.

How that impacted SEDE?

We populate SEDE by using views to select from each table we're going to export. One of the tables we export is the CloseAsOffTopicReasons. The change to the Close UI allowed a column to hold values that were null, but the import was looking for non-nullable columns - these don't play nicely with each other and the import went boom.

We updated the view for the CloseAsOffTopicReasons to no longer contain the nullable column, and actually added some new columns to it. The new list of columns for the table in SEDE is:

  • Id
  • IsUniversal
  • InputTitle
  • MarkdownInputGuidance
  • MarkdownPostOwnerGuidance
  • MarkdownPublicGuidance (initially called MarkdownPrivilegedUserGuidance)
  • MarkdownConcensusDescription (sic)
  • CreationDate
  • CreationModeratorId
  • ApprovalDate
  • ApprovalModeratorId
  • DeactivationDate
  • DeactivationModeratorId
Improve this answer
4
  • 4
    Nice! So the actual trick for getting more data in SEDE (e.g. a 'user is a moderator' indicator) is 1) ask for a related feature request 2) make sure it gets implemented in a way that breaks SEDE 3) ... 4) profit?
    – Glorfindel Mod
    Commented Apr 28, 2020 at 18:26
  • 6
    Don't get any ideas. :P
    – Taryn StaffMod
    Commented Apr 28, 2020 at 18:30
  • 2
    Thank you, Taryn! I suppose "we" should update the list in meta.stackexchange.com/a/2678/307535 too?
    – Jeff Schaller
    Commented Apr 28, 2020 at 18:32
  • I've updated the Database schema documentation in the link above. Taryn, please accept my thanks (and pass them on to Nick and anyone else involved) for handling this today and not "off-hours". I support a good work-life balance! :)
    – Jeff Schaller
    Commented Apr 29, 2020 at 1:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .