I’m definitely not the type to evangelize something I don’t think is useful. Ask friends of mine, and they’ll likely tell you how I went through phases where I endlessly promoted things like Notepad++ for Windows, Netbeans IDE (PHP), Sequel Pro, Gnome-Do, Thinkpads, Macbooks, Toy Story 3, iPod Touches, and Visual Studio ’08. I just can’t help it. When I get excited about something, I have a hard time stfu-ing.

But holy crap. Let me tell you — if you think the WordPress default search sucks — as I did 2 years ago, try WPSearch. It’s (IMO) the second-most-useful plug-in that has ever existed for WordPress. It also has the slickest admin UI I’ve ever used in WordPress (a big thanks to JQuery and it’s plug-ins, of course).

I wrote WPSearch, and I 100% recommend it to anyone running something that isn’t a traditional WordPress blog. Run a recipe catalog? Use It! Review engine? Use it! Shopping engine? What the hell are you doing without my plug-in on your site? Your customers can’t find shit!

Fun scenario: Let’s imagine you sold laser printers on your site along with other computer peripherals, and every product page on your site had “[brand] [model] Printers” in it’s title, like “Dell 1100 Laser Printer”. If someone searched for “printers” with default WordPress search, they wouldn’t get those products back.

Does that scare you? It should. Get WPSearch now. And if you don’t like it, tweet @_kennyk_ and be brutally honest. I can take it.

Side note #1: When I wrote WPSearch, I was also trying to cash-in big time at a programming contest where the grand prize was a trip to RailsConf and the runner-up would get a set of steak knives (presumably to stab whoever won the grand prize).

Side note #2: Toy Story 3 is an intellectually provocative and introspective masterpiece!

Side note #3: Askimet is #1, but is tragically taken for granted

If you want to read about the background of the WPSearch Search plug-in for WordPress, read below. But if you just want the gist of this post, here it is:

WPSearch is the best search plug-in for your WordPress blog. It is a stemming, stop-word blocking, fast, relevant, fulltext search for WordPress. There isn’t a single plug-in in the WordPress repository that can do what it does.

You can get it here: http://wordpress.org/extend/plugins/wpsearch/ — or just install it through the WordPress plugin administration backend. Just search for WPSearch.

If you run any sort of monetized blog, you might be losing sales or readership if users can’t find what they need on your site. According to Adotas, 43% of users hit the search box first.

Background:

I’ll admit that I’ve been pretty vocal already about WordPress’ lacking search functionality. I’ve been vocal about it for 2 years, and it was the original impetus to get this blog up and running. It was also the reason that I wrote wpSearch, the original version of WPSearch — the lucene-based search plug-in for WordPress. wpSearch was the first true fulltext search for WordPress, in my opinion.

But the problem with wpSearch was that it wasn’t highly engineered. I had written it for a programming contest, and I was on a tight deadline — and when projects are rushed, the quality of code goes down. And because of that, bugs made their way out of the woodwork over the next two years.

Consumed by college work and my job, I didn’t have much time to address those issues. In fact, I had declared wpSearch unsupported a year after it was released in 2009.

A year following that, Daniel Hay at Pixelberry in New Zealand had requested that I add the ability of searching within a category for a client of his. That’s when WPSearch 2 development began.

I wanted to give wpSearch a full rewrite, and correct several mistakes I made with the first version:

• Rename it from wpSearch to WPSearch
• Change the listing name in the WordPress repository to WP Search, so searches for ‘search’ would bring it up
• Follow and MVC pattern (it’s a complex plug-in)
• Build a configurable search driver framework, so any driver could be written to search and index
• Build better logging
• Evangelize it to no end (wpSearch was barely promoted)
• Give it a UI independent of the default WordPress stylesheets, and also make it stylish

The biggest change is the configurable driver part. The free version of WPSearch contains a driver that uses Zend Lucene in the background. Any driver, however, can be written to work with WPSearch. Drivers for SOLR, sphinx, the Google Search Appliance, or name-your-own-search-product could be written.

I did that because PHP is not the best language to write a search engine in. Since Zend_Search_Lucene is the backend driver of the free version, there is an upper-bound of scalability on the plug-in. After all PHP is a scripting language, and I doubt Zend ever really imagined someone would make the ludicrous decision of indexing tens of thousands of posts in PHP. I found the breaking point to be about 20,000 docs. At that point, I ran into memory issues, slow mid-indexing optimizations, and slow first-hit (non-cached) searches.

So the point is that I poured everything I had into WPSearch 2, and I want to tell everyone about it. I did this project under the umbrella of OConf, my start-up, with business partner John Crepezzi. John’s an ex-engineer at Sun Microsystems, and he spends his days at Patch.com now. He also wrote the backend driver for WPSearch Pro, an alternate driver for WPSearch 2 which can handle up to 500,000 docs.

John and I gave a talk at Wordcamp NYC, where we officially launched WPSearch 2.The topic was on the default WordPress search, and why avoiding a remedy to it can lose you both readers and money. If you run a shopping engine, people can’t find your products. If you run a news site, readers can’t find your content.

If you don’t think people even use the search box — heads up, the advertising gurus at Adotas say 43% of users who find your site do.

You can check it out here: http://wordpress.org/extend/plugins/wpsearch/ . In it’s short time in the repository (about 9 days), it’s already had around 1,000 downloads, and I’ve had a lot of positive feedback coming in. I also dropped a comment about it on WordPress founder Matt Mullenweg’s site, where he mentioned a search product for Drupal. Hopefully he’ll check it out and let me know what he thinks.

Check it out!

After an exhausting week and a half tracking down the source of a mysterious bug in wpSearch, I think I can finally close the book on the “null result” issue that had me pouring over the source code.

wpSearch 1.5.0.5, the first official release after the 1.5 landmark, brings to the forefront some of the features and fixes slated in the last post. wpSearch 1.5 has had the following features implemented:

• Comment Searching
• A behind-the-scenes event logger for easily figuring out user issues
• An upgrade to the underlying Lucene Search
• An upgrade to the underlying StandardAnalyzer (used for relevancy)

And these fixes:

• No more null results after a post is edited
• Foreign character support (or simply indexing content with ‘UTF-8′ encoding
• Memory issues for content-heavy posts

wpSearch 1.5.0.5 is a rock-solid release that is starting to make a name for itself in the WordPress world. The new ‘Phone Home’ feature in wpSearch allows users to report their copy of wpSearch. A few of the blogs with wpSearch currently in use are listed here:

• Building The Ergonomic Guitar
• ComputerBob
• Savoring Kentucky

Patrick Cushing at the EnterVenture blog wrote a very detailed comparision of the default WordPress search’s relevancy vs. wpSearch’s. This article ended up at digg.

Of course, as far as wpSearch has come in its short lifespan, there exists a set of users that deserve credit for pointing out issues and keeping me informed of bugs, needed features, etc.. So, in no particular order, I would like to thank:

• ComputerBob, at ComputerBob.com for pointing out the first instance of the empty result issue. He has thoroughly documented his usage with wpSearch at his blog, in a fair and balanced fashion. Furthermore, he has sent his index data back with detailed comments when most users would simply give up on wpSearch. Thanks ComputerBob.
• Robert Irizarry, who has kept the wpSearch thread at the WordPress repository stuffed with feature ideas and issue notices.
• Olivier, who’s 6000 posts provided the first failed scalability test for wpSearch. His pointing out of this issue led to a change to allow for greater scalability — in other words, wpSearch 1.5 was tested successfully up to 7,000 posts. Great dedication to detailing these issues has helped wpSearch greatly.
• Karl Heigl, who first mentioned the fact the wpSearch was not handling German accents, and subsequently all foreign (to the U.S.) characters. This also ended up affecting Olivier. This bug was fixed in 1.5.0.5. Thanks Karl!
• A user named Brian, said, “Thanks for the update. If you need any other information or even help testing, I’d be happy to assist. Just let me know. ” Thanks for your support Brian.
• And to all those who have donated to this project so far!

So, wpSearch 1.5.0.5 wouldn’t be at it’s current status if it weren’t for those supporting it.

Features coming up for wpSearch include result highlighting, contextual snippets, and a progress meter for index building.  I encourage everyone who is reading this but hasn’t installed wpSearch yet to try it out, and see the awesome blog search that you’ve been missing.

wpSearch (more info in my previous post), the lucene-powered search plugin for WordPress, has officially been accepted into the WordPress plugins repository. You can view and download wpSearch here:

http://wordpress.org/extend/plugins/wpsearch/

The latest version as of right now is 1.1.0.0. Several major features have been added since the original beta release.

• Seamless integration of wpSearch into your blog. After you activate wpSearch and build your blog’s search index, the search box on your blog will now be configured to use wpSearch for searches.
• You can now decide whether you want search results in the page ( the standard ), or have them loaded into and AJAX search pop-up. (Originally, the AJAX pop-up was the only way to view results ). This option is configurable via the WordPress admin screen.
• Bloggers can now tweak the importance of things such as title, content, and tags in a blog search. This effectively allows control over what is considered relevant in a blog search.

So what’s next for wpSearch?

More searchable content: It’s no secret that the best content on a blog is sometimes in the comments. This is especially true for bloggers of tech and programming sites where blog readers often put useful contributions in comments.

The opening of the source: At SourceForge! Sure, PHP is inherently open-source (it’s a scripting language, after all!). But the best future for wpSearch would entail its placement into SourceForge.NET where the coding community can have the opportunity to contribute to the wpSearch project. wpSearch is already registered at SourceForge, and has a project page at:

http://wpsearch.sourceforge.net/. (Right now, there isn’t much setup up).

I plan to have wpSearch developed at SourceForge, and have stable releases be uploaded to the plug-in repository at WordPress.

There are some other features I plan to add to wpSearch very shortly, one of which is contextual search result content, so you can see the words around the matching content of a search result.

I can’t think of the others off the top of my head. What I would really like to know is if anyone finds wpSearch to be of value so far, and whether they are having any difficulties.

I read on another blog that blogs get xx% more comments if the words “Have your say” are at the end of a post. I think I’ll try that.

Have your say!

There are many things I love about WordPress — the extendability, the ease of use, and large library of themes available online, to name a few. But if there is one aspect of WordPress that needs a little work, it is the default search functionality.

Recently, I’ve been spending a lot of time working on a search plugin for WordPress that is based on the Lucene search engine — a very cool and powerful search library used by a lot of big places. The plugin is in its beta stage, and ready for use and evaluation by anyone who would like to check it out. The plugin is currently implemented on my blog, so you can use the search box on the upper-right side to see it in action.

wpSearch uses the PHP port of the library by Zend. It also spawned a sub-project, the PHP StandardAnalyzer. You can read more about that here.

The search currently uses a lightbox floating over the page to allow users to navigate search results. An option to integrate the results into the page may be and option in the future.

The major features of wpSearch are:

• Unmatched and customizable search relevancy (that’s the power of Lucene working)
• Very fast search speed
• Wildcard and Boolean operator support
• Easy installation
• Instantly updated searching after a post has been written
• Searching of Posts and Pages

Features for advanced users:

• Customizable interface via CSS
• Access to the internal search service for extendability

wpSearch was written for a development contest at LTech Consulting (a firm specializing in search with Lucene and the Google Search Appliance), but with the full intent of being open source.  If anyone is interested in helping develop it, drop me a comment on this post.

Also, if anyone gives this plugin a try and has any suggestions, I would really appreciate your input! Just leave a comment and I’ll get back to you. The plugin will be made available in the WordPress search repository shortly.

Full information about wpSearch (installation instructions, screenshots, etc) is available on wpSearch’s project page.

In my last post I spoke a little about Zend’s Lucene implementation in PHP, and its extensive usefulness for content-oriented PHP web applications. One of the roadblocks to implementing a Google-like search, however, was the absence of a stemming analyzer in the Zend package.

While using PHP Lucene, I came across this issue while developing a plug-in for wordpress. I wasn’t getting the relevancy I needed in test searches that I was looking for, and I decided to develop one of my own. I decided that this analyzer should:

• Stem words for greater search relevancy
• Use the pre-existing Zend lowercase filter
• Filter out a standard set of stop words using the Zend stop words filter

After a couple days working on the issue, I’ve developed an analyzer that performs these tasks. I’ve named it the ‘StandardAnalyzer’ after the implementation Java’s Lucene has. You can download the StandardAnalyzer at its project page.

Just a few notes on the about its creation:

• It is not meant to sit within the Zend framework folder. The ‘StandardAnalyzer’ should sit alongside it, and is configured accordingly. The reason for this is to keep what is Zend’s in Zend’s folder, and what is the user’s in his own. I figured that if the StandardAnalyzer was ever integrated into framework, the good folks at Zend would know best how they would like it.
• The code provided handles English words only, but organized to encourage future languages as well.
• I must give a special thanks to Richard Heyes, whose Stemming algorithm is used instead of my own. In tests, I found his code to be a bit more elegant and quicker than my own, which was a direct port of the Java stemming algorithm. From what I gather, Richard is a Zend-Certified Engineer, making his code usage very fitting.

Example Usage

I’ve decided to pack the StandardAnalyzer with an example project and index to make things a little easier for those looking to use it. The example project, as well as most user projects, would start off like:

require_once 'Zend/Search/Lucene.php';
require_once 'StandardAnalyzer/Analyzer/Standard/English.php';

As mentioned before, the StandardAnalyzer folder should sit in the same directory as the Zend Framework. Now that you have the power of Zend ready to go, you can proceed build your index. But don’t forget that to use the StandardAnalyzer, you have to set the default analyzer to an instance of the StandardAnalyzer. So before you index documents or search over the index, you should call:

Zend_Search_Lucene_Analysis_Analyzer::setDefault
( new StandardAnalyzer_Analyzer_Standard_English() );

I folded that line to keep it looking readable.

Anyway, any indexing or searching you do after this line uses the Standard analyzer. (I may not have been very clear, but the same analyzer needs to be used when indexing and searching, or else you won’t get many results.) You can also change the getDefaultAnalyzer() code in Zend/Search/Lucene/Analysis/Analyzer to reference your the StandardAnalyzer too. But I would rather not change this code of the Framework, and leave it in untainted form.

So take a look at the StandardAnalyzer project, and the example project. The example was put together fairly quickly, but it should provide a good example of how to use it. I think a synonym filter would make a nice addition in the future, so I might take a look into that.

Lucene is an open source search engine written in Java. If you have never heard of it prior to now, listen to this: It allows you to create a mini google-like search for anything. That’s right — anything.

But I’ll be a little more specific: Consider you run a news website — or a wiki for that matter. How would you let users search the website? For most programmers, the answer is in implementing a plain-vanilla SQL search over the title and content of the articles. There are a few issues with this approach:

• The search time can be fairly lengthy
• Running a LIKE query can still be very inaccurate (a search for ‘manager’ over a field containing ‘manage’ will not be considered a match)
• There is almost no relevancy relationship in the way the results are ordered

Lucene is a Java package which lets a Java programmer insert documents into an ‘index’, basically the search engine’s data base, and search over that index later on. So it is a true search engine that a Java programmer can use to grab information at incredible speeds: milliseconds in my tests.

The details of Lucene can be found at its Apache incubator Site.

I’ll get to the real point of this post. Considering how useful a tool Lucene would be, you are probably somewhat disappointed that I said it was for Java. After all, many would find something like this most useful if integrated with a server-side language such as PHP.

Zend, a PHP devoted firm most noted for the Zend Framework created a “Search” component as part of its framework, which is a port of Lucene for PHP. Using this port can be extremely useful for implementing search functionality in a web application. There is a single problem standing in the way of creating a true full text search, although, and that is the default search functionality provided in PHP Lucene.

Consider a scenario where we are employers searching for prospective employees on a job search board. In a certain applicant’s resume, he states that he has “Managed a software team with great success, and has great managerial skills.”

Let’s assume this guy’s resume, as well as thousands of other resumes are in a lucene search index. When an employer executes a search on the job board, the job board code than uses the Lucene API to find documents matching the manager’s search terms “sales manager”.

Using the standard functionality of PHP Lucene, our employer would likely never find our mentioned employee. Why? Because the word ‘managed’ and ‘managerial’ is not the word ‘manager’. Even though this document is very relevant to the employer’s search, it will be nowhere within the result set.

Java Lucene has a way to overcome this scenario: the Standard Analyzer. The Standard analyzer is a component that Java Lucene can use to manipulate data when it is going into a search index. So when “Managed a software team with great success, and has great managerial skills” is put in the index, it will be stored as “manag a software team with great success, and has great manag skill.” The standard analyzer performs lower casing and word stemming on the data of a document.

The analyzer is also used on queries. “Sales manager” would become “sal manag”. Now a query of these terms would definitely turn up the employee we just spoke about.

PHP’s Lucene unfortunately does not have this ability yet. My next post will be about my creation of such an analyzer for PHP Lucene.