I don't know why, but my WAG is that indexing and/or returning results for unregistered users was removed for performance reasons. As an optimization it makes a good deal of sense because:
there are likely to be many more unregistered users than registered ones1, and
it seems unlikely that searching for unregistered users will turn up any useful results.
It seems possible that the new search engine will allow performant searches of unregistered users. But will it be worth whatever extra cost that implies? Without knowing the cost-side of the equation (we need a Stack Exchange developer to look into that), we can only consider the possible benefits: why would such a search be useful?
I can think of two use cases:
Find a post from an unregistered user whose name you happen to know.
Perhaps the name was memorable or because you know it from some context outside the site. For instance, I know that Tommy Wasserman, a foremost expert on the New Testament book of Jude, provided an an answer on Biblical Hermeneutics. However, a search of users turns up nothing.
Find all the unregistered users with the same name.
When you think about it, this is a sort of subset of the first case. I've noticed a fellow on Biblical Hermeneutics who starts a new unregistered user to ask a question or two before taking an extended break. He goes by Radz Brown or something. Again, the name does not turn up in the user search or even a full text search of the site.
Google to the rescue
Thankfully, Google doesn't ignore unregistered users. So I can find Tommy Wasserman's post (and any posts that reference his work) and all of Radz' unregistered users. Unlike the built-in search, Google doesn't know that I'm just interested in user profiles, so we get all the various places each unregistered user shows up: badge lists, question lists, tagged questions, answer bodies, and etc. But we can narrow the search down:
site:[site_name].stackexchange.com/users "User Name"
That gives us just the users who have that particular name.
Rather than expand the built-in search, perhaps it would make sense to link to the Google search when a search for an unregistered user comes up empty.
1. This conjecture is hard to prove without being able to search unregistered users.