Really, really, really TL;DR:
This has been fixed.
TL;DR:
The source of the problem was a schema change* to allow NULL values for TagSynonyms.OwnerUserId. Previously, we internally weren't able to disassociate a (say, deleted) user from being the owner of a tag synonym, because we couldn't change the column value to NULL. This change (in combination with data changes in the meantime) caused some of the databases in the SEDE process to fail. The two databases observed here were Literature and Drones, but there were others.
* I joked in the comments about Friday afternoon releases, but had this specific change been shipped on Monday morning or Tuesday at lunch, we still would have had the same result over the weekend.
Background:
The process of populating the SEDE databases is that in the production database we have a set of views, one for each table we copy over to SEDE. We use views because we don't necessarily need all of the columns for any given table, or we want to change the name of the table (e.g. PostsWithDeleted), or the order of the columns, or what have you. So normally the process looks like this:
As I've written about recently here, changes to underlying tables can cause the views that reference them to have stale / inaccurate metadata. Usually this involves adding, dropping, or renaming columns, or changing the data type, and usually it is also accompanied by views that erroneously use SELECT *. In this case, the view does not use SELECT *, and the only thing that changed was whether the column allows NULL (from no to yes), a less common change, but still vulnerable. So when the SEDE process ran and generated a new table based on the metadata of the view, the view still thought the column was not nullable, so the generated destination table had that column defined as NOT NULL:
When it started loading data from the source table, though, in a few databases, it encountered NULL values, and raised this exception (which made a lot more sense after unpeeling the entire onion):
Attempting to set a non-NULL-able column's value to NULL
In the previous version of the code, this failure halts the job, and it goes no further. I've updated the code to skip past any such failures, which allowed the job to process the rest of the databases to minimize the delay. I corrected the problematic ones in the meantime, by running sp_refreshsqlmodule against the TagSynonyms view in the source database; then, I processed those ones again manually, and this time the destination column allowed NULL. (As an aside, this manual part is why Drones is the newest database, instead of the usual StackOverflow.)
Nobody involved with reviewing the PR (myself included) anticipated the downstream effect on SEDE. We've certainly added it to our internal "lessons learned" documentation.
I've got discussion items queued up for how we'll avoid this in the future, including whether we should just refresh all of the SEDE views after any schema change or, more simply, prior to the weekly SEDE process.
