What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 127 Stack Exchange communities.

This question now has a completely different answer, given the announcement that Lucene.NET is being used to power StackOverflow's search capability.

With that, I've created a new question to request a fleshing out of the details and linked the two.


As per the entry on the StackOverflow blog titled "SQL 2008 Full-Text Search Problems" there is a strong indication that Lucene.NET is being used for a number of the search operations on the back end.

So my question is, how is it integrated?

Of course, there are many aspects to the how of it, so I'll try to fill in the blanks as to what I think as best I can.

Indexing

Questions - I believe that this is in an index of its own containing a unique id to quickly look it up based on a Term instance of that id (indexed, not analyzed).

Tags - These are kept in a separate index with multiple fields which have the ids to the documents in the Questions index. Or, if that's too large, there is an index with just the tags and another index which maintains the relationship between the tags index and the questions they are applied to. This way, when you click on a tag (or use the URL structure), it's easy to see in a progressive manner that you only have to "buy into" if you succeed:

  • If the tag exists
  • Which questions the tags are associated with

Replies - Again, I would think this is in a separate index of its own, with a key back into the Questions index.

How are you doing it? What fields are you storing, indexing, analyzing (all operations can be separate operations, or mix-and-matched)? Just how much are you indexing?

Are you using special stemmers/porters?

Boost

This is related to indexing of course, but I think it merits it's own section.

Are you boosting fields and/or documents? If so, how are you boosting them? Is the boost constant for certain fields.

For example, in the document, does the title get a boost over the body? If so, what boost factors have you seen work well?

Most importantly are you using vote data to apply a boost to content (such as questions and answers)?

If so, how relative is the vote data to the search results?

Search

When actually performing a search, all of the indexes will have to be utilized for separate queries.

  • When querying for a tag (by specifically clicking on one or using the URL structure for looking up tagged content), that's a simple TermQuery against the tag index for the tag, then in the associations index (if necessary) then back to questions, Lucene.NET handles this really quickly.

  • When doing a general phrase or term search against content, what and how do you integrate other information (such as votes) in order to determine the results in the proper order? For example, when performing this search on asp.net mvc, these are the tallies for the top five results (when using the relevance tab):

    q votes answers accepted answer votes asp.net highlights mvc highlights
    ------- ------- --------------------- ------------------ --------------
     21          26                    51                  2              2
     58          23                    70                  2              5
     29          24                    40                  3              4
     37          15                    25                  1              2
     59          23                    47                  2              2
    

Note that the highlights are only in the title and abstract on the results page and are only minor indicators as to what the true term frequency is in the document, title, tag, reply (however they are applied, which is another good question).

How is all of this brought together? The biggest questions that stand out are:

  • How is vote data weighed against the relevance score (and what boosts are applied to that score, either directly in the query, in the indexing, or indirectly by multiplying the relevance factor by a result of the vote data).

  • If the replies are indexed separately from the questions, how are those relevance scores brought together? My initial thoughts on this would is that the relevance score in the question is added to the relevance score of each individial reply and whichever pair has the highest relevance is the relevance score for the entire question and all answers.

  • What are the factors outside of analysis that affect all of this? How does vote/view/accepted answers/etc/etc affect the outcome?

Of course there are a lot of other questions I have about this, but as the answers start to frame things, I can expand upon that.

share|improve this question
    
Please consider removing the portions of the question that are duplicated in the SO question, and not relevant since Lucene is not used. (ie, turn it into a simple "Is lucene used or not?" question") Also, putting the answer in the question is generally frowned upon. –  Adam Davis Feb 19 '10 at 19:13
    
@Pollyanna: i think he put the answer in the question because the answer was in a comment so it couldn't be accepted. –  Kip Feb 19 '10 at 22:22
    
@Pollyanna: Kip is right, I put the answer in the question because it was in fact put as a comment. If an answer was added by him as the definitive answer, I'd be more than happy to accept it and remove the comment at the top. As far as changing the question, the questions on SO and here are different in flavor, and changed to reflect the difference between "how would one do" (on SO) and "is it" (here). Changing it doesn't change my impressions or observations at the time it was asked, and if anything, serves as information one might consider when using Lucene.NET applied to the SO model. –  casperOne Feb 19 '10 at 23:52
add comment

5 Answers

up vote 4 down vote accepted

As of Jan 27th, 2011, stackexchange is using Lucene.NET.

You can read Jeff Atwood for the details.

http://blog.stackoverflow.com/2011/01/stack-overflow-search-now-81-less-crappy/

share|improve this answer
add comment

It's not being used yet. It's all SQL queries. They were able to push back the need for Lucene by doing things like removing the 1000 most common words from search queries and some other tricks like that. The ungoodness of the site search is kind of a running joke here.

share|improve this answer
    
@Kip: Unfortunately, the there's no evidence to support that Lucene.NET isn't used. While I have been following the "crappiness" of the searchability of the site, I haven't seen anything yet to suggest that Lucene.NET isn't behind it (only the link to the post I've referenced to indicate that the schedule was moved up aggressively). Just because one pops in Lucene.NET doesn't mean that search improves automatically, it doesn't work like that. –  casperOne Feb 19 '10 at 15:56
1  
If they actually implemented Lucene search, I'm quite certain Jeff would mention it on the SO blog and on the podcast and Joel would probably mention it on his blog too. On the podcast Joel has complained about the lack of Lucene and Jeff has basically said he doesn't think they need it that bad yet. I know he mentioned in this talk that there was no Lucene: blog.stackoverflow.com/2009/05/… (granted, that was 10 months ago, but that was also 5 months after the post you link to) –  Kip Feb 19 '10 at 16:54
6  
Lucene.NET is not being used for Stack Overflow - we are using SQL Server Full Text indexing. Search is an area where we continue to make minor tweaks. –  Geoff Dalgas Feb 19 '10 at 17:05
    
@Geoff - Consider adding that as an answer... –  Adam Davis Feb 19 '10 at 18:31
add comment

No, the site does not use Lucene at all - see the comment from Geoff Dalgas, one of the site developers on this answer:

Lucene.NET is not being used for Stack Overflow - we are using SQL Server Full Text indexing. Search is an area where we continue to make minor tweaks.

share|improve this answer
    
@Pollyanna: Again, that's not really the question (i.e. "How bad is the site's search"). I've shown a link where the statement was made that the timetable for using Lucene.NET for in-site search is was being moved up agressively. I'm looking for whether or not that's been done, and if so, how. If you could cite the policy, it would help, because then a timeline can be established, and we can weigh in more on the question of whether it's been done or not. –  casperOne Feb 19 '10 at 16:07
1  
@casperOne - I've edited my answer to be more clear. I'm not going to search around for references for you. In the absence of referenced answers, you might put some stock in the idea that many of the answers here agree, and that the people answering have been around a long time and should have some idea of whether it's been implemented or not. –  Adam Davis Feb 19 '10 at 16:49
1  
I think the timeline was pushed up to "6-8 weeks": meta.stackexchange.com/questions/19478/the-many-memes-of-meta/… :) –  Kip Feb 19 '10 at 17:03
    
Here's a related post about it: meta.stackexchange.com/questions/10934/… –  Adam Davis Feb 19 '10 at 17:04
add comment

The extensive list of tools used on SO in the FAQ here seems to indicate that Kip is indeed correct when stating that Lucene.Net is not being used.

Jeff has always made a point to list every product used to built SO with and I know they also have Brent around to help with the more complex SQL problems. He also spent some time optimising the databases for SO. At the time Lucene was looked at as an option but they have not implemented it since.

share|improve this answer
add comment

When they were looking at lucene, I understood it was largely driven by performance issues with the native sql server full-text indexing rather than correctness or relevance issues. I believe those performance issues were resolved, and therefore lucene was never implemented.

Since they added the user:me, [tagname], intitle:1, and other "operators", I've been content with the search here.

share|improve this answer
    
The switch to SQL server 2008 resolved the performance issues - text search was brought inside the engine, rather than the crufty add-on it appeared to be in the previous version of MS-SQL. They still had issues with the 2008 SQL server, but over time they resolved them. –  Adam Davis Feb 19 '10 at 17:01
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .