Re^4: Is there a way to compare strings without using an array?

[Jeri]

I like the List::Compare. It looks good. I'm just worried about, because my array will be at least 5 million in length. Can you offer me any peace of mind?

[tj_thompson]

Abstract your comparison code out into a subroutine and give it a try. Should be quick simple code to write and you'll know exactly how well it performs. Will make changing the code or comparing it to other possible solutions simple as well.

[CountZero]

Here is some code which does what you want:

use Modern::Perl;
use List::Compare;

# Make two arrays with fake data
my @left  = map {'AA' . int(rand (5000000) + 1000000)} (1 ... 5000000)
+;
my @right = map {'AA' . int(rand (5000000) + 1000000)} (1 ... 5000000)
+;

my $start = time;
my $lc = List::Compare->new('-u', \@left, \@right);

{
    open my $OUT_INTER, '>', './intersection.txt';
    say $OUT_INTER $_ for $lc->get_intersection;
}
{
    open my $OUT_DIFF, '>', './difference.txt';
    say $OUT_DIFF $_ for $lc->get_symmetric_difference;
}
my $duration = time - $start;

say "Duration: $duration seconds.";

However, on my small laptop (only 2 GByte RAM and running a lot of other programs and services) I get an "Out of memory" error, so two lists of 5 million proteins is too big for it. YMMV if you have a more performant machine. It runs fine and fast for 500,000 proteins (15 seconds), 1,000,000 proteins (52 seconds) and 2,000,000 proteins (185 seconds).