Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re: memory use when merging tied hashrefs

by Eily (Monsignor)
on Nov 13, 2019 at 13:33 UTC ( #11108629=note: print w/replies, xml ) Need Help??


in reply to memory use when merging tied hashrefs

It might depend on which version of perl you are using, but one way to avoid using a temporary list (%$hash1, %$hash2 is flattened to a list of pairs before being added to the anonymous hash) you can iterate over the hashes using each.

while (my ($key, $value) = each %$hash1) { $hash3{$key} = $value; }
Same for $hash2. This also depends on how the tied hash is implemented but it's worth a try...

Edit: or you could also tie hash3 to fetch data either from hash1 or hash2 ... really depends on what you are trying to achieve and what are your contraints.

Replies are listed 'Best First'.
Re^2: memory use when merging tied hashrefs
by Anonymous Monk on Nov 13, 2019 at 14:50 UTC
    you can iterate over the hashes using each

    Nice! Memory doesn't budge when using each, but the program's slightly too slow, so I'll trade memory for speed. Good to know for a cloudy day. I did away with hash3 and merged hash2 into hash1:

    while (my ($k,$v) = each %$hash2) { $hash1->{$k} = $v } 
    
    Thanks for the lesson

      Another possible middleground solution is to use a for loop on the keys. $hash3->{$_} = $hash1->{$_} for keys %$hash1; This potentially means that only the keys will be stored in a temporary list (not sure about the exact impact though, even more so when tied hashes are involded).

      Also, to improve the speed, you can force perl to preallocate a hash with keys %hash = 200;, that's a little bit tricky to use because the right value is the number of buckets in the hash, not the number of keys. See the end of Scalar values about that syntax. To choose the correct number, you can use Hash::Util::bucket_ratio() after doing the job once and getting an idea of the expected size of the hash (do that on the final hash, not a tied one though!). Without that step, perl may first allocate a hash that is too small, and need to copy it into a bigger one when you add too much data. I doubt this happens often with "only" 8000 keys, but it's always worth a try.

      (Thanks to choroba for helping find the relevant documentation :D)

      If $hash1 is tied to a database, does that mean you are adding the records from $hash2 into the first db?

        If $hash1 is tied to a database, does that mean you are adding the records from $hash2 into the first db?

        Yes. That was dumb. Next customer! :-)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11108629]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (5)
As of 2020-05-27 10:41 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    If programming languages were movie genres, Perl would be:















    Results (154 votes). Check out past polls.

    Notices?