Re: KinoSearch & Large Documents

[creamygoodness]

3MB shouldn't be any trouble at all. Indexing time increases roughly linearly with the length of the text. (Once KinoSearch's internal caches start getting flushed the numbers get noisy, but that happens every 10-15MB of content and it's unlikely to be the problem.)

Here's a benchmarking result demonstrating the relationship:

$ perl time_to_add_doc.plx
          s/iter   size_4M   size_2M   size_1M size_500k size_250k
size_4M     16.4        --      -51%      -77%      -87%      -94%
size_2M     8.00      105%        --      -53%      -73%      -87%
size_1M     3.73      340%      114%        --      -41%      -72%
size_500k   2.20      646%      264%       70%        --      -52%
size_250k   1.06     1450%      656%      253%      108%        --
$

One possibility is that that somebody, somewhere has used one of the hateful match variables: $' $` and $&. Their appearance anywhere in your script or its dependencies will completely destroy the performance of KinoSearch's Tokenizer, which runs a short regex over a large string many times in a tight loop.

It's shocking how awful things get, and indeed, the degradation is geometric. Check out Devel::SawAmpersand for an explanation and the sawampersand function which you can use to investigate. Old versions of Text::Balanced are known to cause this problem. (Maybe I should have Tokenizer's constructor issue a warning if Devel::SawAmpersand::sawampersand() returns true.)

Another possibility is that you're hitting swap. It sounds like you've got adequate RAM on that box, and KS itself doesn't need a whole lot -- 30MB or so, not factoring in space occupied by the current doc. But it's worth my asking about for the sake of completeness, since the symptoms are consistent with that diagnosis, too.

If neither of those help, try running the script I used to generate the benchmarking output above. If it produces results in line with mine, that suggests that the problem lies elsewhere in your app.

#!/usr/bin/perl
use strict;
use warnings;

use KinoSearch::InvIndexer;
use KinoSearch::Analysis::PolyAnalyzer;
use Benchmark qw( cmpthese );

my $invindexer;
my $one_k_of_content = 'xxx ' x 256;

sub refresh_invindexer {
    undef $invindexer;
    my $analyzer
        = KinoSearch::Analysis::PolyAnalyzer->new( language => 'en' );
    $invindexer = KinoSearch::InvIndexer->new(
        analyzer => $analyzer,
        invindex => 'test_invindex',
        create   => 1,
    );
    $invindexer->spec_field( name => 'content' );
}

cmpthese(
    5,
    {   size_250k => sub { refresh_invindexer; add_a_doc(250)  for 1 .
+. 3 },
        size_500k => sub { refresh_invindexer; add_a_doc(500)  for 1 .
+. 3 },
        size_1M   => sub { refresh_invindexer; add_a_doc(1000) for 1 .
+. 3 },
        size_2M   => sub { refresh_invindexer; add_a_doc(2000) for 1 .
+. 3 },
        size_4M   => sub { refresh_invindexer; add_a_doc(4000) for 1 .
+. 3 },
    }
);

sub add_a_doc {
    my $repeat  = shift;
    my $doc     = $invindexer->new_doc;
    my $content = $one_k_of_content x $repeat;
    $doc->set_value( content => $content );
    $invindexer->add_doc($doc);
}

[TedYoung]

You may have hit the nail on the head. Even without Devel::SawAmpersand, a quick search of my 250,000 line codebase found several $&'s. That's bad! But I didn't realize it was that significant. I will update my OP with the results. Thanks.