Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re^2: Memory efficient way to deal with really large arrays?

by salva (Canon)
on Dec 13, 2020 at 18:18 UTC ( #11125129=note: print w/replies, xml ) Need Help??


in reply to Re: Memory efficient way to deal with really large arrays?
in thread Memory efficient way to deal with really large arrays?

That makes me thing that one can actually calculate the counters without using any memory at all. You only have to sort the array (or arrays) and then traverse it counting consecutive equal elements.

... well, obviously, unless you need to keep the counters in memory for some further processing, but as you have said, the OP hasn't told us yet what his final goal is!

  • Comment on Re^2: Memory efficient way to deal with really large arrays?

Replies are listed 'Best First'.
Re^3: Memory efficient way to deal with really large arrays?
by LanX (Cardinal) on Dec 13, 2020 at 19:01 UTC
    If you keep the pairs, i.e. 2 dim points, how would you want to sort them in one dimension?

    > the OP hasn't told us yet what his final goal is!

    Exactly!

    For instance, if he wanted to process the data sequentially he could also sort them into HoHs or AoHs, since Perl is swapping out unused data.

    Like this only the currently processed second level structure would be in RAM and the performance overhead for swapping would be negligible.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11125129]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (4)
As of 2021-04-15 08:11 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?