KinoSearch + Ferret = Lucy

[creamygoodness]

Greets,

Dave Balmain, author of Ferret, a port of the Java Lucene search engine library written in Ruby/C, and I are joining forces. Doug Cutting, original author of Lucene, is on board as an advisor and a tie-breaking vote.

The project, named Lucy, will be a loose C port of Lucene, with Perl and Ruby bindings. Development is being hosted at Apache.

If everything goes as planned, at some point a "Lucy" CPAN distribution will be released and will eventually replace my loose port of Lucene to Perl/C, KinoSearch. KinoSearch is not going away -- it will continue to be actively maintained at least until it's certain that Lucy is sufficiently feature-rich, mature, and stable to take its place. And although we are starting out fresh with an empty directory, KinoSearch's back end is serving as our template -- as spelled out in the original project proposal -- so it would be fair to characterize the transition as KinoSearch morphing into Lucy. (I'll give Lucy a mention during my upcoming talk at OSCON, Building Search Applications With KinoSearch, but I'm not changing my slides. :) )

Our alliance solves what had been a problem for both KinoSearch and Ferret: they were primarily the work of a single author, and developing and maintaining a port of Lucene is a lot of work. We expect development to progress rapidly, since the primary unsolved engineering challenge is integration, and that's much, much easier than what we've already accomplished.

If you'd like to either contribute or follow the project's progress, mailing list subscription information can be found at http://lucene.apache.org/lucy/mailing_lists.html.

[rinceWind]

I guess this will ultimately supersede CLucene and Plucene, no?

[creamygoodness]

Not in the same sense, because I don't control those modules. Assuming that Lucy reaches maturity, I'll release a "deprecated" version of KinoSearch that points people to its successor.

In another sense, one could say that KinoSearch already has superseded Plucene, as it's about 25x faster for indexing at least. Note that KinoSearch owes a debt to the Plucene team, as will Lucy, for having yielded significant insights into why porting Lucene to other languages has been so difficult.

The CPAN hooks for CLucene are bare-bones. There's been talk of a forthcoming CPAN distro under the name "Lucene" which would provide much more extensive bindings for CLucene (not Java Lucene, which is confusing, but there you go)... but nothing has been released yet.

In summary, if you're looking for a Lucene-alike on CPAN today that's both speedy and full-featured, KinoSearch is probably what you want. Sometime in the future, KinoSearch will quietly yield to Lucy. The API and the file format will be slightly different (and therefore incompatible), but changing over will be pretty painless.

[jdporter]

Can you give a status update on the Lucy project? It appears to be stalled. :-(

Also, do you know anything about what's going on with JSAN? Doesn't seem to be much action over there, either.

Thanks....

[creamygoodness]

Nope, not stalled. But the progress is not transparent, so I can see why you would think that.

The short answer is that because Dave Balmain became unavailable for a while, I chose to work with the existing KinoSearch community and code base while implementing many of the ideas for Lucy.