Strange memory growth

spica1001 has asked for the wisdom of the Perl Monks concerning the following question:

Hi all. Here's an odd one, to me anyway. I'm running a script to process very large XML files with embedded JSON. When I run it, the memory increases indefinitely (the file can be 100s of GB and the RAM usage reaches 25GB+). Boiling it down, I reach the below. If I remove the line marked "## THIS LINE", the memory remains static, but leave it in and it increases again. Adding in a load of undefs seems to make no difference. It's evidently leaving the hash array around, but I can't see how to 'free' it.

Why would accessing a non-existent hash value cause that, or of course even better how do I prevent it?! Thanks for any help...

use JSON;
open(IN,"<:utf8","$ARGV[0]");
while(<IN>)
{
    if (m!^\s+<text.*?>({[^\{\|].+})</text>!) {
        my $jt = $+;
        $jt=~s/\&quot;/\"/g;
        my $json = new JSON;
        my $jp = $json->allow_nonref->utf8->relaxed->decode($jt);

        my $c = $jp->{'claims'};
        # "claims":{"P31":[{"mainsnak":{"snaktype":"value","property":
+"P31","hash":"...","datavalue":{"value":{"entity-type":"item","numeri
+c-id":5},...}... }...}],
        if (ref($c) eq 'HASH') {
            foreach my $ch (keys %$c) {
                if (ref($c->{$ch}) eq 'ARRAY') {
                    foreach my $cg (@{$c->{$ch}}) {
                        if (defined $cg->{'mainsnak'}->{'datavalue'}->
+{'value'}->{'notexist'}) {}  ## THIS LINE
                    }
                }
            }
        }
    }
}
[download]

====================================================

UPDATE: Thanks to all the replies, I've worked it out now.

Points should go to tinita as the suggestion of use strict pointed me in the right direction. It turned out that, in some of the lines that my script reads, the value $cg->{'mainsnak'}->{'datavalue'}->{'value'} is a string, not a hash. It seems treating a string as a hash causes the memory growth. I fixed it with:

if (ref($cg->{'mainsnak'}->{'datavalue'}->{'value'}) eq 'HASH' && defined $cg->{'mainsnak'}->{'datavalue'}->{'value'}->{'notexist'}) {}

(Of course my code does plenty else besides this, but the principle of needing to check that a variable is indeed a hash before checking for a key is the main takeaway here.)

Comment on Strange memory growth Select or Download Code

Replies are listed 'Best First'.
Re: Strange memory growth by ikegami (Patriarch) on Feb 14, 2018 at 18:26 UTC
In lvalue context, `$var->{$key}` [download] is equivalent to `( $var //= {} )->{$key}` [download] so `defined( $cg->{mainsnak}{datavalue}{value}{notexist} )` [download] is equivalent to `defined( ( ( ( ( $cg //= {} )->{mainsnak} //= {} )->{datavalue} //= {} + )->{value} //= {} )->{notexist} )` [download] That could potentially create a lot of new hashes. Solution 1: `if ( $cg && $cg->{mainsnak} && $cg->{mainsnak}{datavalue} && $cg->{mainsnak}{datavalue}{value} && defined( $cg->{mainsnak}{datavalue}{value}{notexist} ) ) { ... }` [download] Solution 2: `no autovivification; if (defined( $cg->{mainsnak}{datavalue}{value}{notexist} )) { ... }` [download]	[reply] [d/l] [select]
Re^2: Strange memory growth by tinita (Parson) on Feb 14, 2018 at 18:50 UTC
That could potentially create a lot of new hashes. But only three, and $cg (and $jp) fall out of scope after every iteration. I can't see why this would continually increase memory.	[reply]
Re^3: Strange memory growth by ikegami (Patriarch) on Feb 14, 2018 at 18:54 UTC
Indeed. As such, I find it hard to believe the line you identified has anything to do with a memory leak. Is that the program you actually ran?	[reply]
Re^4: Strange memory growth by tinita (Parson) on Feb 14, 2018 at 18:56 UTC
Re: Strange memory growth by huck (Prior) on Feb 14, 2018 at 18:28 UTC
Try this `if ( defined $cg->{'mainsnak'} && defined $cg->{'mainsnak'}->{'datavalue'} && defined $cg->{'mainsnak'}->{'datavalue'}->{'value'} && defined $cg->{'mainsnak'}->{'datavalue'}->{'value'}->{'notexist +'}) {}` [download] see http://search.cpan.org/~vpit/autovivification-0.18/lib/autovivification.pm	[reply] [d/l]
Re^2: Strange memory growth by spica1001 (Initiate) on Feb 19, 2018 at 14:58 UTC
Thanks all for the excellent responses. To answer the various suggestions and questions: Yes, I posted the exact script that I'm running. Perl version 5.22.1, Ubuntu 16.04. Trying huck's or ikagami's suggestions didn't change things, unfortunately. Running on the same line repeatedly shows no memory growth, which I can't explain. It's thus possible it's only certain lines of the input file that cause the problem but I'm struggling to see how to identify which. Thus it's difficult for me to post data on which I see the problem. The file I'm reading in is the Wikidata data dump (33GB bzipped) https://dumps.wikimedia.org/wikidatawiki/20180201/wikidatawiki-20180201-pages-articles.xml.bz2 (I usually read it in with `IO::Uncompress::Bunzip2` but verified that the issue still occurs when unzipped by taking the first 2GB of that file unzipped) I'll work on trying to get `Data::Diver` working and identify problematic lines of the input file.	[reply] [d/l] [select]
Re: Strange memory growth by tinita (Parson) on Feb 14, 2018 at 18:42 UTC
Which perl version are you using, which OS? I don't see a specific problem with the code. It should have a `use strict;` at the beginning, though. How long is a typical JSON string? Can you create a sample input that will show memory increase? For example, instead of reading in the file, just loop over the same line a couple of 1000 times and watch memory consumption. The problem the others are mentioning doesn't look like a problem to me since $jp falls out of scope after every loop iteration. Memory stays the same for me if I run this 500,000 times on an example line. edit: specifically, how many array elements does a typical structure have? my assumption was that you have many lines, but maybe your lines/JSON strings are very big?	[reply] [d/l]
Re^2: Strange memory growth by Anonymous Monk on Feb 16, 2018 at 14:50 UTC
I'm pretty sure that auto-vivification is the root-cause problem here, because a chain of hashrefs was used in a single expression, causing unwanted AV to occur from left to right. And I am also pretty sure that huck had the best answer.	[reply]
Re: Strange memory growth by Marshall (Canon) on Feb 15, 2018 at 06:36 UTC
I updated my post with some more test cases. defined and exists behave similarly in regards to generating intermediate keys in the process of doing their work. I show an example of how to prevent this below. I don't know if this helps or not, but checking for defined vs or exists can generate hash entries. #!/usr/bin/perl use strict; use warnings; use Data::Dumper; my %hash = ('key' => 'value'); print Dumper \%hash; #prints: #$VAR1 = { # 'key' => 'value' # }; print "defined\n" if defined $hash{abc}{xyz}; # will create new keys print "defined\n" if defined ($hash{x}) and defined ($hash{x}{y}); # a +dded: won't create new keys print "exists\n" if exists ($hash{x}) and exists($hash{x}{y}); # a +dded: won't create new keys print "exists\n" if exists $hash{x1}{x2}; # added: will create new +keys print "defined\n" if defined $hash{x3}; # added: no new key print Dumper \%hash; __END__ #prints: # The "abc" key is created due to the check for defined of 2nd dimensi +on # exists will also create new intermediate keys $VAR1 = { 'key' => 'value', 'abc' => {}, 'x1' => {} }; [download] Sometimes you have to check at each level if the hash "dimension" exists. A check for "defined" or "exists" can generate automatic dimensions. ~~Sometimes you have to you have to check whether some hash key exists at all before checking what its value is (defined or not).~~	[reply] [d/l]
Re^2: Strange memory growth by Anonymous Monk on Feb 15, 2018 at 13:29 UTC
So are you saying that Huck's post should have used exists, instead of defined? To be absolutely clear, can you post a quick example similar to Huck's? Auto-vivification is obviously the root cause problem here ... but are you saying that defined will auto-viv too? (That does seem counter-intuitive, but ...)	[reply]
Re^3: Strange memory growth by Anonymous Monk on Feb 15, 2018 at 14:13 UTC
No difference between `exists` and `defined` in this regard. In both cases, the intermediate levels will spring into existance. The last level isn't dereferenced as a hash. If you have perl installed, you can try it out yourself: `$ perl -e 'use Data::Dump; exists $a->{foo}{bar}; dd $a;' { foo => {} } $ perl -e 'use Data::Dump; defined $a->{foo}{bar}; dd $a;' { foo => {} }` [download]	[reply] [d/l] [select]
Re^4: Strange memory growth by Anonymous Monk on Feb 16, 2018 at 14:46 UTC
Re^5: Strange memory growth by Marshall (Canon) on Feb 16, 2018 at 21:48 UTC
Re: Strange memory growth (Data::Diver) by Anonymous Monk on Feb 17, 2018 at 20:46 UTC
Classic solution is to use Data::Diver `use Data::Diver qw/ Dive /; ## if (defined $cg->{'mainsnak'}->{'datavalue'}->{'value'}->{'no +texist'}) {} ## THIS LINE if ( defined Dive( $cg, qw' mainsnak datavalue value notexist ' + ) ) { }` [download] Ex `$ perl -e "use Data::Diver qw/Dive/; $f{a}{b}{c}=666; print Dive(\%f, +qw/a b c Q/); print Dive(\%f, qw/a b c/); " 666` [download]	[reply] [d/l] [select]


Perl-Sensitive Sunglasses
	PerlMonks