13

Recently I have seen (and this was confirmed by other users) problems with search.

When searching (on basically any site):

  • sometimes the search takes unusually long, but returns results;
  • sometimes the search returns no results;
  • sometimes the search returns page saying: "We are currently offline for maintenance." (And that StackStatus or the status blog should be checked for updates - but at the moment I do not see any recent posts there.)

Some screenshots can be found in the posts linked below.

As observed by Nog Shine, searching for tags (and words that are automatically converted to tags) seems to work fine.

What is actually the problem? Will this be fixed?

This seems to be a network-wide issue - it has already been reported on some per-site metas:

Improve this question
0

1 Answer 1

Reset to default
11

We had a cluster member register a voltage spike on the 10G NIC this morning which caused it to drop from the cluster ungracefully, causing a rapid reallocation and queue exhaustion which caused all other errors downstream (thread_pool.search.queue_size, specifically).

I am still working on cluster stability now, we'll need a few more node moves to tweak settings to prevent it from recurring. I don't have an ETA at the moment, but will post here if we can figure that out (ES is a bit erratic on recovery times with our workload).

Update 11:50 UTC: I have disabled search on Stack Overflow to speed recovery of the cluster (we were very unstable as the alternative).

Update 12:10 UTC: Cluster has stabilized and search has been re-enabled network-wide. Since the majority of the impact came from our API search endpoints, we'll be deploying separate kill switches to relieve load there without taking site search offline for most users. We'll continue to monitor the cluster.

Improve this answer
4
  • 3
    Would someone be kind enough to translate "cluster member" and "queue exhaustion" and "node moves" and "ES" and other bits of jargon into English for outsiders like me? Thanks.
    – Gerry Myerson
    Oct 23, 2018 at 11:13
  • 2
    Cluster: group of cooperating computers. Cluster member: one computer in that group. Queue exhaustion: when a queue (of tasks) becomes too long and malfunctions (either stops accepting new tasks or drops existing tasks). ES: ElasticSearch. I don't know what Nick means by node moves though.
    – Erik A
    Oct 23, 2018 at 11:27
  • 1
    @Erik that's one of the deuterium fusion reactors needed to get the impulse engine up and running again
    – Petter Friberg
    Oct 23, 2018 at 12:07
  • 5
    Nick - got some other issues that may/may not be related but definitely suspicious... can you have a look at meta.stackoverflow.com/questions/375674/… and then have a look at these SO search results and if you look back about an hour or so - there's a strange amount of duplicated/deleted questions even with the same title - one of 'em 8 times in under a minute which shouldn't be allowed and quite a spattering of others that looks sus...
    – Jon Clements
    Oct 23, 2018 at 12:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .