This question now has a completely different answer, given the announcement that Lucene.NET is being used to power StackOverflow's search capability.
With that, I've created a new question to request a fleshing out of the details and linked the two.
As per the entry on the StackOverflow blog titled "SQL 2008 Full-Text Search Problems" there is a strong indication that Lucene.NET is being used for a number of the search operations on the back end.
So my question is, how is it integrated?
Of course, there are many aspects to the how of it, so I'll try to fill in the blanks as to what I think as best I can.
Questions - I believe that this is in an index of its own containing a unique id to quickly look it up based on a Term instance of that id (indexed, not analyzed).
Tags - These are kept in a separate index with multiple fields which have the ids to the documents in the Questions index. Or, if that's too large, there is an index with just the tags and another index which maintains the relationship between the tags index and the questions they are applied to. This way, when you click on a tag (or use the URL structure), it's easy to see in a progressive manner that you only have to "buy into" if you succeed:
- If the tag exists
- Which questions the tags are associated with
Replies - Again, I would think this is in a separate index of its own, with a key back into the Questions index.
How are you doing it? What fields are you storing, indexing, analyzing (all operations can be separate operations, or mix-and-matched)? Just how much are you indexing?
Are you using special stemmers/porters?
This is related to indexing of course, but I think it merits it's own section.
Are you boosting fields and/or documents? If so, how are you boosting them? Is the boost constant for certain fields.
For example, in the document, does the title get a boost over the body? If so, what boost factors have you seen work well?
Most importantly are you using vote data to apply a boost to content (such as questions and answers)?
If so, how relative is the vote data to the search results?
When actually performing a search, all of the indexes will have to be utilized for separate queries.
When querying for a tag (by specifically clicking on one or using the URL structure for looking up tagged content), that's a simple TermQuery against the tag index for the tag, then in the associations index (if necessary) then back to questions, Lucene.NET handles this really quickly.
When doing a general phrase or term search against content, what and how do you integrate other information (such as votes) in order to determine the results in the proper order? For example, when performing this search on asp.net mvc, these are the tallies for the top five results (when using the relevance tab):
q votes answers accepted answer votes asp.net highlights mvc highlights ------- ------- --------------------- ------------------ -------------- 21 26 51 2 2 58 23 70 2 5 29 24 40 3 4 37 15 25 1 2 59 23 47 2 2
Note that the highlights are only in the title and abstract on the results page and are only minor indicators as to what the true term frequency is in the document, title, tag, reply (however they are applied, which is another good question).
How is all of this brought together? The biggest questions that stand out are:
How is vote data weighed against the relevance score (and what boosts are applied to that score, either directly in the query, in the indexing, or indirectly by multiplying the relevance factor by a result of the vote data).
If the replies are indexed separately from the questions, how are those relevance scores brought together? My initial thoughts on this would is that the relevance score in the question is added to the relevance score of each individial reply and whichever pair has the highest relevance is the relevance score for the entire question and all answers.
What are the factors outside of analysis that affect all of this? How does vote/view/accepted answers/etc/etc affect the outcome?
Of course there are a lot of other questions I have about this, but as the answers start to frame things, I can expand upon that.