10

Just now search is broken again, here on MSE at least. It takes an incredibly amount of time before returning 0 results.

This very well could be a regression of Search is extremely slow

9
  • Works instantly for me here, what site was this on?
    – Mark Kirby
    Nov 30, 2020 at 9:25
  • Here on this site
    – Luuklag
    Nov 30, 2020 at 9:25
  • 4
    It's working on and off here. For me it timed out few minutes ago, now it's working as usual. Something is/was broken for sure though, so the report is valid. Nov 30, 2020 at 9:25
  • I've tried using the search here for several minutes and in every attempt it gave back the results in a matter of seconds. Seems to be okay again... Nov 30, 2020 at 11:05
  • 4
    @mindstormsboi until next time it'll break. Sadly I'm serious, this is the third time in short while search has major issues. Developer said "we suspect a hardware issue at this point", but I suspect it's hacking attempts or at least DoS attacks causing this, based on the frequency of the "breaks". Hardware can't possibly fail every two weeks. Nov 30, 2020 at 11:07
  • @Shadow10YearsWizard that's some wild speculation right there :p. Digging in a little now - we saw a spike in HTTP requests to the cluster 4 hours ago and that resulted in thread pools backing up.
    – Dean Ward
    Nov 30, 2020 at 14:29
  • Good your looking into this @DeanWard
    – Luuklag
    Nov 30, 2020 at 14:38
  • 1
    @DeanWard well... "spike in HTTP requests" in my glossary is DoS attack. Thanks for the update! Nov 30, 2020 at 15:09
  • 2
    I didn't say it was HTTP traffic to the external load balancers... A DoS perhaps, but appears to be one of our own making... Still digging here!
    – Dean Ward
    Nov 30, 2020 at 15:49

1 Answer 1

14

As has been mentioned in the comments, we've had a fair number of issues with our search cluster over the past few weeks - from DDOS to hardware issues. It's certainly in a more stable state than it was after that bunch of fun, but it's still a little delicate and, as you saw this morning, it had a bit of a blip. Of the ~16.5k search requests within a 20 minute period from 09:05 UTC roughly 4.5k failed - that's about 27% of search requests.

We've done some digging and see that there was an increase in connections to the cluster resulting in thread pool backlogs but this does not correlate with an increase in external traffic - that 16.5k is pretty consistent for the time of day.

That makes it point towards a background process that is inadvertently gobbling up JVM threadpool threads - things like related question rebuilds (those in the right sidebar), although they typically don't have anywhere near enough concurrency to cause such issues. It's also possible that something in Elastic (inter-node communication) caused something to backlog, but we can't be sure (there are a lot of logs to chew through). We're still digging into the root cause but I'm afraid it's not as simple to discover what that is as you might think!

In terms of the future state of all this - we're in the process of provisioning more search nodes, upgrading to a more recent version of Elasticsearch and actively testing optimizations to how we run the cluster to make it more stable. We'll also be adding additional monitoring to ensure we can more effectively diagnose what's happening.

For now, I can't guarantee that the issue won't happen again in the short-term, but we are keeping an eye on things to try and keep it all as stable as we can while we make fixes.

1
  • 2
    Thanks for this extensive summary of what happened, and how you are trying to make this work. Seeing all the recent troubles I wonder if Elasticsearch is still the way to go, although I am not hindered by any knowledge of search "engines" in this question.
    – Luuklag
    Nov 30, 2020 at 19:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .