Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re^2: memory use when merging tied hashrefs

by Anonymous Monk
on Nov 13, 2019 at 14:50 UTC ( #11108633=note: print w/replies, xml ) Need Help??


in reply to Re: memory use when merging tied hashrefs
in thread memory use when merging tied hashrefs

you can iterate over the hashes using each

Nice! Memory doesn't budge when using each, but the program's slightly too slow, so I'll trade memory for speed. Good to know for a cloudy day. I did away with hash3 and merged hash2 into hash1:

while (my ($k,$v) = each %$hash2) { $hash1->{$k} = $v } 
Thanks for the lesson
  • Comment on Re^2: memory use when merging tied hashrefs

Replies are listed 'Best First'.
Re^3: memory use when merging tied hashrefs
by Eily (Monsignor) on Nov 13, 2019 at 17:18 UTC

    Another possible middleground solution is to use a for loop on the keys. $hash3->{$_} = $hash1->{$_} for keys %$hash1; This potentially means that only the keys will be stored in a temporary list (not sure about the exact impact though, even more so when tied hashes are involded).

    Also, to improve the speed, you can force perl to preallocate a hash with keys %hash = 200;, that's a little bit tricky to use because the right value is the number of buckets in the hash, not the number of keys. See the end of Scalar values about that syntax. To choose the correct number, you can use Hash::Util::bucket_ratio() after doing the job once and getting an idea of the expected size of the hash (do that on the final hash, not a tied one though!). Without that step, perl may first allocate a hash that is too small, and need to copy it into a bigger one when you add too much data. I doubt this happens often with "only" 8000 keys, but it's always worth a try.

    (Thanks to choroba for helping find the relevant documentation :D)

Re^3: memory use when merging tied hashrefs
by Eily (Monsignor) on Nov 13, 2019 at 15:56 UTC

    If $hash1 is tied to a database, does that mean you are adding the records from $hash2 into the first db?

      If $hash1 is tied to a database, does that mean you are adding the records from $hash2 into the first db?

      Yes. That was dumb. Next customer! :-)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11108633]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (4)
As of 2020-07-05 19:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?