Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re: Memory efficient way to deal with really large arrays?

by salva (Canon)
on Dec 13, 2020 at 11:37 UTC ( #11125100=note: print w/replies, xml ) Need Help??


in reply to Memory efficient way to deal with really large arrays?

Instead of using an array of arrays for storing the pairs, you can use two arrays, one for the first elements and another for the second ones. That is going to reduce your memory consumption by an order of magnitude:
my @d; my (@p1, @p2); for (my $i=0;$i<500000000;$i++) { my ($p1,$p2) = getPair(); $d[$p1]++; $d[$p2]++; push @p1, $p1; push @p2, $p2; }

Replies are listed 'Best First'.
Re^2: Memory efficient way to deal with really large arrays?
by LanX (Cardinal) on Dec 13, 2020 at 13:01 UTC
    That's brilliant. It's simple and very efficient.

    Another approach would have been to pack the pairs into a 64 bit string to be stored in one array, that might be more efficient but wouldn't be as simple as this trick.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

      that might be more efficient

      I am not sure. Some years ago Nicholas Clark introduced a change in the internal representation of SVs that made then very efficient for integers and doubles. Briefly, all internal types used to have a fixed size head (SV_HEAD) and a variable sized body. That change, used a trick to embed the body inside the header for SVs representing numbers, making then more cache friendly and reducing memory usage by eliminating the body part.

      So, having two arrays with numbers may actually use less memory than an array of small strings.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11125100]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (7)
As of 2021-04-20 16:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?