Re: Re: Slurping BIG files into Hashes

[waswas-fng]

Looks like you have something goofy going on there. look at the time report at the bottom of this post for my runtime on a 2 proc sun box.

open (CONFIG, "<iaout.txt") || die "Coulnd't open config file!";

my %lookup;

while (<CONFIG>)
{
    $lookup{substr($_, 0, 13)} = substr ($_, 13);
}


#script used to generate data in the form of:
# 21 random alpha chars per line
#
#
#use Data::Random qw(:all);
#open IA, ">iaout.txt";
#for $x (1 .. 160000) {
#my @random_chars = rand_chars( set => 'alphanumeric', min => 21, max 
+=> 21 );
#print IA @random_chars, "\n";;
#
#}[1:50pm] 161 [/var/tmp]: time perl t
2.95u 0.18s 0:03.30 94.8%

How simular are the key parts of the data in your file? I am wondering if you are getting a very high collision rate on they key for some reason? either that or memory is my best guess.

-Waswas

[Elgon]

Aha, t'was written...

"I am wondering if you are getting a very high collision rate on they key for some reason?"

I reckon that this is what the problem is as the key values are very similar all the way through and there's not a lot which can be done about it. Hmmm... I'm trying to think of a better data structure. Thanks for the help everybody who contributed.

Elgon

PS - The box is an 8 processor Sun server running Solaris with 8GB of RAM. Neither the IO nor the memory seem to be the problem from continuous observation of the stats.

update - Thanks to BrowserUK et al. for their help unfortunately the version we are using is 5.004_5 and I am not allowed to change it. Oh well. I'm trying to find a workaround as we speak...

update 2 - Thanks to jsprat, the script now runs in about a minute. Ta to all...

Please, if this node offends you, re-read it. Think for a bit. I am almost certainly not trying to offend you. Remember - Please never take anything I do or say seriously.

[jsprat]

Try presizing the hash - keys %lookup = 160_000;

If it is hash collisions, this might solve the problem.

dominus has an interesting bit at perl.plover.com called When Hashes Go Wrong.

Update: Meant to ask you to "print scalar %lookup;" after all is done. scalar %hash will give you the number of used buckets / number of allocated buckets. If the number of used buckets is low (like 1/16) all your hash items have been put in the same bucket!