27

How does Stack Overflow implement search indexing? Does it index the questions and answers as soon as the user posts them or will it run a batch job for indexing content (questions and answers). Does it have a single index or multiple indexes?

I know it uses Lucene.net for searching and indexing, since writing into Index is a IO operation, how does it manage the writing part?

Please shed some light into indexing policy at Stack Overflow?

0
30

This isn't really a dupe of A new search engine for Stack Exchange since it's asking about implementation so I'll give it a go.

Our methodology to most things is this:

  1. do it simple
  2. if dead simple doesn't work then make it only as complex as it needs to be to function
  3. optimize as much as is warranted (is it rare, is the user waiting on it? etc. are considered)

Search, in my view, remains pretty simple in it's implementation. We are now on top of elasticsearch, but this pattern has been around since before that with slight differences:

  1. Grab all posts that changed since the last index pass
  2. Index them

How do we do this?

Posts is a table in our database; it contains both answers and questions. We added a LastChange column of the rowversion data type. What this does is add a timestamp column (which has nothing to do with time, think of it as just a big 8 byte number) that increments every time the row is updated (a shared sequence between all rows).

The approach is simple, get all the posts and index them the first time (in parallel, lots of threads, etc.) keeping track of the max rowversion from what we selected. When done, store that rowversion (as an int) in the index itself as a separate document type - that way the state of the index is totally contained in the index, and relies on nothing else to be in sync to behave properly. A practical example is a dev downloading a fresh prod database...a simple delta index "just works" here because of the design.

On a delta indexing pass (every index pass after the first), we simply Select <stuff> From Posts Where LastChange > Cast(@LastChange as timestamp) (where @LastChange is the int we stored in the index earlier, just grabbing it from the index at the start of the pass). We then use the same index code to parallel index those - that's it. We run this process every n seconds, where n depends on the site.

Every site has a database, that's 1:1 with every site having an index to match that database. Using elasticsearch that also allows us to do a /_all/ across all sites at once (hint, hint).

Currently, this process runs every 30 seconds on stackoverflow.com and here, 5 minutes for all other main sites, and 10 minutes for child metas.

I won't go into where it runs, because that's actually a set of services on a set of service boxes that make our app zip itself, transfer, and setup a parallel and switching app domain set inside that service - it's...a little complicated. You don't need that, wherever this pattern runs is fine, a route you hit, a service you run, whatever works in your environment is juuuuuust fine.


What will the setup be this time next year? Who knows, likely event-based indexing though. Just like realtime updates you see to your rep, many things in our system have triggers and events that fire for easier code and flexibility at the same time. As we get events on everything, we may switch to on-edit and on-creation indexing for all sites. We could use the existing system as a backup - re-indexing doesn't hurt anything as long as we merge or purge every so often.

Off the top of my head, I'd say an event fires on the C# side, this would trigger a redis message publish. Those same services on the service tier as above would be subscribed to that channel and would just listen for the Post.Id to index...it would grab from SQL and index it.

This setup would allow realtime indexing lagging only a fraction of a second, without delaying you (the user) at all, and if it failed for any reason, the delta index on an interval would clean it up. We make heavy use of redis pub/sub already (e.g. someone upvotes, you see your rep change) - so the code for this is also pretty simple.

6
  • Thank you nick(I'm glad I got an answer from the guy who implemented the elasticsearch for stackexchange). I got the point, i will try to implement Nest for my test mvc project on similar lines. Lets see where I land. I won't go for Redis for now. – kewlguy Jul 5 '13 at 23:36
  • Are the 30 second SO, 5 minute other sites, 10 minute meta sites update intervals described here still accurate at this time? – Jason C Mar 30 '15 at 13:28
  • 2
    2 years later, I'm curious how/if this system has changed – jonallard Jun 18 '15 at 13:56
  • @jonallard Since the search is still using the same backend I don't think anything has changed. – Shadow Wizard Wearing Mask V2 Jan 21 '16 at 15:37
  • Oh @Nick and 30 months later, is this scenario working well? – Shadow Wizard Wearing Mask V2 Jan 21 '16 at 15:38
  • I was mainly asking because it hints at some changes "this time next year" – jonallard Jan 21 '16 at 18:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .