Re^2: Ampersands and sub speed

[creamygoodness]

Perl/XS search engine library which uses the Java Lucene file format. Java Lucene uses method calls for everything down to writeByte(), so a big part of making things work in Perl is moving speed critical code into XS loops. But sometimes there's just no getting around calling a function or a method.

[revdiablo]

But sometimes there's just no getting around calling a function or a method.

And you're trying to run more than 1.4 million method calls per second?

Update: I guess running one method call per byte, this would limit you to 1.4MB/s. Which isn't really all that bad, but could be a problem, I guess. I think it would be more fruitful to redesign the code not to call a method for every byte than it would to try and use a faster type of call, though.

[creamygoodness]

Say you have an index with a million documents in it. Someone searches for a term that hits 300,000 of them. Calculating the score for each one of them in Java Lucene might involve several tens of method calls. Now imagine a search with 5 terms. This stuff adds up.

[Errto]

Just fyi, you should definitely check out Plucene if you haven't yet. Even if you can't use it directly (since the binary format of the index is not the same as Java Lucene), the code may be useful since they presumably have to solve the same problem you do.

[creamygoodness]

Good advice. I've been heavily involved in the Plucene project. This is like Plucene, but the indexer, at least, is 13 times faster, and it writes true Lucene-compatible indexes. I'm working on the search code.