Search and Unicode characters

Question

Apparently, searching for Unicode characters in SE sites is supposed to work. But I found a bug which manifests on French Language & Usage.

• FL&U contains two posts with the word sœur (notice the œ): 1, 2.
• If I search for "soeur" (without the œ), both are found
• If I search for "sœur" (with the œ), only one is found.

But there is more than that. Let's look at the word œuf. There's at least one post that includes it. Yet, if you search for either œuf or oeuf, it finds nothing.

So, something is clearly going wrong :)

Answer 1

Your searches now behave appropriately since the move to elastic, you can test them here:

• sœur
• œuf

Answer 2

I get the two expected search results for both the œ and the oe sœur variants, so that seems to be working fine. Edit: Like Gilles, I now get zero results for soeur, and I have no explanation for that.

The issue with your second search has to do with word boundaries. The search does whole-word matching, so a search for œuf or oeuf will turn up nothing, because the actual word used is œufs. Searching for œufs yields two results, although still not the one you were looking for.

Why is that? Well, in keeping with the issue of word boundaries, it seems that the search considers words containing apostrophes to be single units, likely because this makes sense for English contractions. So, searching for "d'œufs" (with quotations, otherwise it strips the apostrophe) will in fact turn up the question containing the answer you were looking for*.

So, there's really two issues here, and if they can be addressed, I'm all for it, although I identified some potential issues:

1. Plural forms aren't considered to be matches of singular forms
   • Plural forms are complex, so this probably isn't doable in a non-sub-word matching sort of way, or at all (without direct mapping) for some languages
2. Apostrophes aren't considered word boundaries, which likely makes sense for most of the sites, but is problematic for the French site (and potentially other non-English sites as well)
   • This might not be controllable on a per-site level

_{*One of the results for œufs by itself does match on d'œufs, but this is because the text uses a different apostrophe that is considered a word boundary.}