Normally, Lucene stems all docs as they go into the index. During searches, Lucene stems any words in the search phrase.

Really, that’s the extent of word stemming.

I think I may have an idea of what’s going on. Since all documents which are added to the search results are scored, that score is normalized to a value between 0 and 1. I’m thinking that in search #1, since you have an exact match between the search phrase and the stemmed word, the score is 1.00 while other matches get a very low score (and arent added to the final result list). In fact, “mortage” and “mortga” are considered completely different words in the post-analysis operations.

I could be wrong, but I think it makes sense. Your situation might be an interesting edge case in the StandardAnalyzer search.

And I think “mortgagee” to “mortgage” might be incorrect, but I’m not sure of whether the Porter Stemmer was ever considered anything other “the best available,” so I guess we’ll have to settle

‘mortgagee’ (as in person that borrows a mortgage) stems down to ‘mortgage’, I was expecting ‘mortgag’

Is this a bug or correct behavior? Here’s more to the story (I am using zend_search_lucene):

———————–

I’m using Porter Stemmer to stem the words, and here’s a problem I’m running into:

Word “mortgage” is correctly stemmed to “mortgag” Word “mortgagee” is (arguably incorrectly) stemmed to “mortgage”

There are approximately 100 documents with the word “mortgage” There is 1 document with word “mortgagee”

When I build an index without putting “mortgagee” in any documents, everything works fine: searching for “mortgage” or “mortgages” or “mortgag” returns all 100 documents.

When I build an index and one of the documents contains “mortgagee”, searching the index for “mortgage” only returns a single document with “mortgagee” (which was stemmed down to “mortgage”). However, searching for “mortgag” or “mortgages” returns all 100 documents.

The only logical conclusion I can make from this problem is lucene first searches for the pre-stemmed word, and if it doesn’t find any results, it continues to search for the stemmed word. Thus, when searching for ‘mortgage’, it first finds the ‘mortgage’ that was stemmed from ‘mortgagee’ and stops searching. Is this the correct behavior, or is it a bug?

$this->addFilter(new Zend_Search_Lucene_Analysis_TokenFilter_LowerCaseUtf8());
$this->addFilter(new Zend_Search_Lucene_Analysis_TokenFilter_StopWords($this->_stopWords));
$this->addFilter(new StandardAnalyzer_Analysis_TokenFilter_EnglishStemmer());

I am only able to add filters whic are of type Zend_Search_Lucene_Analysis_TokenFilter. Lets say if I try to add $this->addFilter(new Zend_Search_Lucene_Analysis_Analyzer_Common_TextNum_CaseInsensitive) it doesn’t work because its not of type Zend_Search_Lucene_Analysis_TokenFilter. Can you please have extra functionality to cater for this or if you have any solution can you post it or email me.

Thanks alot for your help in advance

Thank you very much.

Peleg.

Maybe it was due to a fault or maybe due to your decision, but my comment here (from last week) had been deleted.

Would you be able to explain me the reason for this? And if not — may I ask my question again?

Thanks ahead,
Peleg.

Do you have an alpha / beta version of the WpSearch plugin? There doesn’t appear to be anything on the Project page. I’m in the process of learning how to use lucene and would love to see how it works on my local version of wordpress.

I’m pretty new to all of this so I’d be happy to give whatever credit / donation you were looking for.

Thanks!