20

For years I have been peeved by how external search engines, in particular DuckDuckGo and Google, are often able to find better, even much better, matching questions on StackOverflow than the internal question search. (And I'm assuming the situation is similar on other network sites, hence the post here.) This is still the case today.

Case in point: Searching for ways to iterate over a simple range of integers in C++. There is one "canonical" question about this on SO.

With internal search:

https://stackoverflow.com/search?q=%5Bc%2B%2B%5D+iterate+over+range+of+integers

we fail to find that question and its answers - even on the second page of results and probably at all; but with external search:

https://duckduckgo.com/?q=c%2B%2B+iterate+over+range+of+integers

the top result is the appropriate question.

Naturally, if you know the exact title of the question you have better luck with SE's internal search tool:

https://stackoverflow.com/search?q=%5Bc%2B%2B%5D+loop+over+a+range+of+integers

I would like to ask that more effort be invested in improving search efficacy.

More specifically, I suspect that part of the problem is the difficulty of accounting for concept closeness, or near-synonimity: "looping" vs "iteration", since if we just switch those two words in the example above we get what we want.

8
  • 2
    Or perhaps look into integrating google search as search mechanism?
    – Luuklag
    Jun 2 at 9:02
  • 8
    @Luuklag: 1. That's an implementation detail AFAIAC, I just want the search box to get me better results. 2. I don't like Google, and it's not the only alternative, but whatever.
    – einpoklum
    Jun 2 at 9:38
  • 6
    The original stance always has been: we can't out perform Google and while true to some extent there are specific features in the on site search that don't extend to external search engines. Originally network-wide search was powered by Google but that got moved to the internal elastic cluster a couple of years ago, with the promise of better stemming. I believe some work is done in that respect but I'm sure the capabilities are not exhausted. And that leads to the mishaps you've shown. We need better search to better serve visitors.
    – rene
    Jun 2 at 9:53
  • 5
    @rene: Ok... but I will say that it's not clear that SE can't outperform Google for searching on-site contents, as Google is a general-purpose, all-internet search engine while here one can make all sorts of specific assumptions.
    – einpoklum
    Jun 2 at 9:57
  • 5
    @einpoklum I'm not sure how a search engine exactly works but I can imagine that you might want to balance index size and performance. SE is running on their own hardware. If they screw-up sizing they can't scale out without first ordering servers and put them in the server rack. But this is Nick Craver territory, I better not make many wrong assumptions here. The request you have is valid and I support it, in case that wasn't clear.
    – rene
    Jun 2 at 10:08
  • A search engine that has broad information on what search terms are associated with each other and what other sites link to is going to perform better. Site-specific search engines are pretty much always severely handicapped. I think there are many areas of SE where the return on investment will be much better than improving search, since I think it will require a major investment for a little improvement.
    – Erik A
    Jun 2 at 14:31
  • 1
    What's always bugged me is how the duplicate question search doesn't take into account the answers. I can understand why it was done, but often it's far easier to find a duplicate question by searching for the answer. A duplicate answer isn't always an indication that it's 100% a duplicate, but they're often better written and more easily found using the keywords you'd expect.
    – Kevin B
    Jun 2 at 22:21
  • 3
    'SO search is bad' - to be fixed in 6-8 decades:( Jun 4 at 14:30
8

TL;DR. AND search by default is bad idea.

I tried your original query, it also shocked me how that query was unable to find the question post. Then I spot a relevant thread (suggested by the "Related" section on the right column) and here is the link: A new search engine for Stack Exchange

It has been emphasized that the search engine is doing AND search.

Do you notice your query has a keyword iterate? That keyword does not occur in the post you want to find. So it becomes clear that if you remove the keyword iterate (query), tada! you get the post! Simply because of the search criteria being too strict...

But no wonder using AND clause is kept through the years, there is even a comment praising this choice:

https://meta.stackexchange.com/a/160101/345491

Although I do not use SE search engine often, I am a full-time researcher on search engine, I can tell that because of all the synonyms, typo, stemming etc., having a default AND filter is really a bad idea in this case, obviously the external search engines you mentioned here like DuckduckGo and Google, they all choose to use OR clauses by default, it is the best practice for a reason.

2
  • 2
    When I type a phrase in the search box, I don't expect its words to be used as a threshold filter. I expect to get questions whose text is semantically close to what I typed in, and questions whose text is "textually"-close to what I typed in. That's not an AND search - nor is it an OR search. Or at least, it shouldn't be any of those two things.
    – einpoklum
    Jun 2 at 21:33
  • 4
    @einpoklum Your idea of getting semantically relevant document is the whole topic about IR research, and it is very very well studied. There is model doing Bayesian statistics to find relevant thing by assigning weight to query terms, namely "IDF". It is not simply OR, in fact, by OR all I mean is to allow any keyword to be omitted if there is a level of certainty that a result is relevant. OR is "threshold filter", but just let you know there is a scoring phase after the "OR filter" usually, and that is where the magic happens in search engines, to identify the most relevant results.
    – Wei Zhong
    Jun 2 at 22:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .