TLDR: Can’t we please use stemmers matched to site languages?

According to this post from Nick Craver and this post from Haney, our searches are using the ElasticSearch stemmers. The problem is that stemmers have to be tuned for each particular language being used, or they generate too many false positives and false negatives. We should please use these, because they appear to be available.

Bad Searches

This problem is especially notable in the various non-English incarnations of Stack Overflow:

This is much worse when you have verbs. English gets to piggyback some of its singular–plural verb inflections atop noun inflections (think drive, drives), but more strongly inflected languages cannot do that. It’s not reasonable for people to have to think about all the tenses, aspects, persons, and moods just to find hits for the verb “return” in its many possible forms.

Bad Tags

This same problem also surfaces in creating new tags, because the duplicate-of-singular/plural checker completely misses tags whose shift in grammatical number triggers “more things in other tongues than are dreamt of English philology”.

Because the Spanish, Portuguese, French, German, Italian et cetera Stack Exchange language sites use such tags in those languages, they’re always getting problems with tags that would be easily fixed simply by switching the stemmer to match the target non-English language.



You must log in to answer this question.

Browse other questions tagged .