21

Since Sunday, I expect after the refresh of 03:00 UTC, the Stack Exchange Data Explorer seems to have troubles serving pages to us.

I notice that I'm either logged out frequently, I get the red error box that something went wrong with executing my query or I'm presented with the maintenance page.

Retrying my action and/or refreshing seems to resolve the issue somewhat until the next hiccup.

There are reports from other users that experience the same problems. And some users report no issues. That indicates the intermittent nature of the problem.

I think that SEDE instance needs a hug so it feels happy again. Can someone take the bike-ride to the data-centre and apply the needed love?

Improve this question
4
  • You steal my meta rep :D
    – 4b0
    Aug 18, 2020 at 7:29
  • 2
    I hesitate to post before conformation and dang dang . We care site so no matter who post.
    – 4b0
    Aug 18, 2020 at 7:41
  • 1
    @Shree yeah, we don't need a bug report for every blib and yesterday it seemed somewhat better so I kept the report to chat only. You mentioning it as well was enough evidence that it was/is a broader problem that warrants a post.
    – rene
    Aug 18, 2020 at 8:37
  • 3
    We're investigating it. We took a memory dump from the app and are trying to figure out what needs to be poked.
    – Taryn
    Aug 18, 2020 at 13:00

1 Answer 1

Reset to default
15

TL;DR; This should be fixed in the next 8-24 hours as we had to block a bot scraping us.

Longer version:

I noticed the flapping over the weekend and started investigating yesterday. My first guess was a SQL issue. After some initial digging, I made some server level changes to see if things would stabilize. They did for several hours, but last night I noticed yet another outage, so I grabbed a memory dump for review this morning.

After a whole lot of flapping overnight, Nick and I started reviewing the dumps this morning, well mostly Nick reviewed and I watched. We noticed some threading issues, so we look at the total connections to SEDE to figure out what was hogging all the connections, but nothing really stood out, so we turned to our Traffic Logs and we found the issue.

We initially queried the logs for the last couple of days and saw we were being crawled by Applebot. Crawled is an understatement, we were getting slammed by it.

Here are some stats of traffic from the Applebot and everyone else to give an idea on scale - the outages started on 2020-08-16 between that date to today (2020-08-18) we had:

Applebot  |  Non-Applebot Traffic
---------------------------------
1484480   |  290063

We've added the Applebot the robots.txt to stop the crawling. It'll take a little while for that to take hold, so we might have a little more flapping of SEDE, but we'll be monitoring it and will kick as needed.

Improve this answer
3
  • 9
    Just tell them you won't pay them their app store cut and they'll block you first. Worked for epic :D
    – Journeyman Geek
    Aug 18, 2020 at 15:04
  • 1
    Maybe worth reporting to Apple?
    – Shadow Wizard Is Sad And Angry
    Aug 18, 2020 at 15:17
  • There isn't a more aggressive defence against crawlers?
    – Mast
    Sep 24, 2020 at 9:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .