36

All migrations are currently broken all across the network. I first noticed this on ELU.

What's happening is they get closed for being offtopic as belonging on another site, then locked, but then those migrations are never picked up by the internal job (software agent) that actually migrates them to their destination.

However, it is also happening on the Portuguese language site, and there have been no migrations even from SO or MSE lately either. So it appears to be a full network-wide outage.

Update as of 2022-10-31 03:00:10Z

It appears that we can migrate questions again, but the myriad stuck ones from when we couldn't remain stuck.

1 Answer 1

14

We had a site-to-site post migrations outage from sometime mid-afternoon UTC on October 27 through late morning UTC on October 30 (a little under 3 days). This is now fixed.

The actual outage was caused by some changes made to our internal network configuration. Site-to-site migrations work via API calls between sites that are configured to never leave our data center, and the changes that were made prevented these calls from going through successfully. These changes have been reversed.

This leaves nearly 3 days of site-to-site post migrations that were not performed. We are currently evaluating ways to proceed with going about performing this backfill network-wide, and will update here when we have more information to share.

Update:

  • The Stack Overflow mods manually went through the 8 posts that were up for migration during this outage, and made decisions on them whether to manually migrate or not
  • As for the rest of the network (where there are on average 14 migrations daily for all non-SO sites combined), we have made the decision to not put in a backfill for this. The logic needed for the backfill is actually quite complicated and unreliable (each one would need to be manually checked anyway, especially after some mods have already gone ahead and made decisions on migrate/not for all affected posts), and is not worth the investment of time to implement for a one-time event like this.
  • Thus, for the approximately 50 affected posts across the network during this 3-day period, we will leave it up to mods of individual sites to act on individual affected posts and will not be performing any automated actions. Affected posts will stay closed-as-off-topic on their originating sites (and mods can manually override this and migrate them if they so choose).
5
  • 5
    How many migrations are there to backfill? Oct 31 at 15:15
  • 1
    @ShadowTheKidWizard If SO is any indication, not very many. We had 8 (all now handled manually, where "handled" includes a moderator deciding it was not worth migrating).
    – Ryan M
    Nov 1 at 6:24
  • @RyanM thanks, so it's not very clear what we waiting for? Nov 1 at 8:31
  • 2
    @ShadowTheKidWizard part of the issue, I suspect, is that some of the posts have been manually reposted, so simply migrating them may create a bunch of duplicates (I messed this up with one of them). And even if it's just one or two per site, that could still be a sizeable chunk of total posts to go through. Alternately, it may just need someone to find time to do it manually. In the case of SO, I was chatting with a CM and we decided ad hoc to just manually resolve it on the spot - but doing that for every site is...a lot.
    – Ryan M
    Nov 1 at 8:34
  • 4
    Would it be possible for someone to let the sites in question know what needs backfilling, assuming they missed it?
    – Journeyman Geek Mod
    Nov 10 at 10:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .