Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re: creating and managing many hashes

by Gtforce (Sexton)
on Feb 19, 2018 at 09:39 UTC ( #1209465=note: print w/replies, xml ) Need Help??


in reply to creating and managing many hashes

An update: I spent the day reading-about/playing-with hashes, and decided to use them to the problem I had.

Satisfied with the results I now have at a constant 20% single-core utilization (approx), and a peak 1.5GB memory footprint.

$>perl pairs_v3.pl Distinct products: 2083 Number of pairs (incl. duplicates): 4,336,806 Number of pairs (excl. duplicates): 2,168,403 Started processing at 14:04:49 Finished processing at 14:07:24 Finished writing to disk at 14:07:43 $>

PS: This is to determine the pairs only (haven't run the stats using the pairs yet which is the next step).

Replies are listed 'Best First'.
Re^2: creating and managing many hashes
by poj (Abbot) on Feb 19, 2018 at 10:27 UTC
    This is to determine the pairs only

    That should take seconds, not minutes.

    #!/usr/bin/perl use strict; my $t0 = time; my @f = map{ sprintf 'product_%04d',$_ } 1..2083; my $count = 0; open OUT,'>','pairs.txt' or die "$!"; while (my $x = shift @f){ for my $y (@f){ ++$count; print OUT "$count\t$x\t$y\n"; } } close OUT; my $dur = time-$t0; print "$count in $dur s\n"; # 19s on raspberryPI
    poj

      poj - that is amazing (took 3 seconds)!!

      Will do a post-mortem now, thanks a ton.

Re^2: creating and managing many hashes
by LanX (Cardinal) on Feb 19, 2018 at 10:29 UTC
    I'm afraid, only telling us about "results" without showing code will only frustrate many and won't help you improve your skills.

    Cheers Rolf
    (addicted to the Perl Programming Language and ☆☆☆☆ :)
    Wikisyntax for the Monastery

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1209465]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (4)
As of 2021-06-20 00:34 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What does the "s" stand for in "perls"? (Whence perls)












    Results (93 votes). Check out past polls.

    Notices?