This isn't really a dupe of A new search engine for Stack Exchange since it's asking about implementation so I'll give it a go.
Our methodology to most things is this:
- do it simple
- if dead simple doesn't work then make it only as complex as it needs to be to function
- optimize as much as is warranted (is it rare, is the user waiting on it? etc. are considered)
Search, in my view, remains pretty simple in it's implementation. We are now on top of elasticsearch, but this pattern has been around since before that with slight differences:
- Grab all posts that changed since the last index pass
- Index them
How do we do this?
Posts is a table in our database; it contains both answers and questions. We added a
LastChange column of the rowversion data type. What this does is add a timestamp column (which has nothing to do with time, think of it as just a big 8 byte number) that increments every time the row is updated (a shared sequence between all rows).
The approach is simple, get all the posts and index them the first time (in parallel, lots of threads, etc.) keeping track of the max rowversion from what we selected. When done, store that rowversion (as an int) in the index itself as a separate document type - that way the state of the index is totally contained in the index, and relies on nothing else to be in sync to behave properly. A practical example is a dev downloading a fresh prod database...a simple delta index "just works" here because of the design.
On a delta indexing pass (every index pass after the first), we simply
Select <stuff> From Posts Where LastChange > Cast(@LastChange as timestamp) (where
@LastChange is the int we stored in the index earlier, just grabbing it from the index at the start of the pass). We then use the same index code to parallel index those - that's it. We run this process every
n seconds, where
n depends on the site.
Every site has a database, that's 1:1 with every site having an index to match that database. Using elasticsearch that also allows us to do a
/_all/ across all sites at once (hint, hint).
Currently, this process runs every 30 seconds on stackoverflow.com and here, 5 minutes for all other main sites, and 10 minutes for child metas.
I won't go into where it runs, because that's actually a set of services on a set of service boxes that make our app zip itself, transfer, and setup a parallel and switching app domain set inside that service - it's...a little complicated. You don't need that, wherever this pattern runs is fine, a route you hit, a service you run, whatever works in your environment is juuuuuust fine.
What will the setup be this time next year? Who knows, likely event-based indexing though. Just like realtime updates you see to your rep, many things in our system have triggers and events that fire for easier code and flexibility at the same time. As we get events on everything, we may switch to on-edit and on-creation indexing for all sites. We could use the existing system as a backup - re-indexing doesn't hurt anything as long as we merge or purge every so often.
Off the top of my head, I'd say an event fires on the C# side, this would trigger a redis message publish. Those same services on the service tier as above would be subscribed to that channel and would just listen for the Post.Id to index...it would grab from SQL and index it.
This setup would allow realtime indexing lagging only a fraction of a second, without delaying you (the user) at all, and if it failed for any reason, the delta index on an interval would clean it up. We make heavy use of redis pub/sub already (e.g. someone upvotes, you see your rep change) - so the code for this is also pretty simple.