Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Re^5: memory use array vs ref to array

by BrowserUk (Patriarch)
on Sep 17, 2016 at 21:41 UTC ( [id://1172023]=note: print w/replies, xml ) Need Help??


in reply to Re^4: memory use array vs ref to array
in thread memory use array vs ref to array

You don't need the do if you don't use the value of the last expression.

True. But it's habitual.

Moreover, I tried both ways on a 2GB file and didn't notice any difference in memory consumption:

I'm not familiar with the ins and outs of memory measurement on *nix; but I can demonstrate the difference on Windows.

C:\test>p1 [0]{} Perl> print mem;; 9,432 K []{} Perl> open I, '<', 'test.dat'; my $s; do{ local $/; $s = <I> }; p +rint mem;; 997,784 K []{} Perl> Terminating on signal SIGINT(2) C:\test>p1 [0]{} Perl> print mem;; 9,440 K []{} Perl> open I, '<', 'test.dat'; my $s = do{ local $/; <I> }; print + mem;; 1,986,060 K []{} Perl> Terminating on signal SIGINT(2)

As you can see, in the latter case, the memory assigned to the process is double: ( 997784 - 9432 ) * 2 + 9440 = 1986146; almost exactly the 1,986,060 K measured in the latter case.

The reason is that in the later case, the data is read into an internal mortal temporary scalar; and then copied from there to the named lexical, before the memory attached to the temp is freed. As the allocation is greater than (from memory) 1MB, (on windows at least) such huge allocations are allocated directly from the OS's virtual memory rather than from the process' heap; and then get released directly back to OS.

Which is usually a good thing, but it still means you need to have double the memory available for a short while, and if you are close to the limits, that can blow the process.

Is it possible that large allocations are also freed back to the OS on *nix, and you are measuring after it has been freed?


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^6: memory use array vs ref to array
by choroba (Cardinal) on Sep 17, 2016 at 22:00 UTC
    > Is it possible that large allocations are also freed back to the OS on *nix, and you are measuring after it has been freed?

    I'm not sure where this mem comes from. I tried with Memory::Usage, measuring the consumption with top running with 0.001s delay (was probably more in practice):

    #!/usr/bin/perl use warnings; use strict; use feature qw{ say }; use Memory::Usage; my $file = shift; open my $FH, '<', $file or die $!; my $string; my $mu = 'Memory::Usage'->new; $mu->record('start'); { local $/; $string = <$FH> }; $mu->record('after do'); undef $string; seek $FH, 0, 0; $string = do { local $/; <$FH> }; $mu->record('after no do'); $mu->dump;

    Output:

    time vsz ( diff) rss ( diff) shared ( diff) code ( diff) + data ( diff) 0 17504 ( 17504) 3916 ( 3916) 3328 ( 3328) 1644 ( 1644) + 1060 ( 1060) start 0 1970632 ( 1953128) 1957112 ( 1953196) 3520 ( 192) 1644 +( 0) 1954188 ( 1953128) after do 1 1970632 ( 0) 1957248 ( 136) 3520 ( 0) 1644 ( + 0) 1954188 ( 0) after no do

    Top output:

    ($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord }map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,

      It looks like you've got your "after do" and "after no do" labels the wrong wayaround; but I agree, it looks like there is no difference on your version of Perl on *nix.

      I was running under my default perl5.10; so I re-ran the tests under the latest version I have installed, 5.22 and I see the same results:

      C:\test>\Perl22\bin\p1.pl [0]{} Perl> print mem;; 9,404 K []{} Perl> open I, '<', 'test.dat'; my $s; do{ local $/; $s = <I> }; p +rint mem;; 997,760 K []{} Perl> Terminating on signal SIGINT(2) C:\test>\Perl22\bin\p1.pl [0]{} Perl> print mem;; 9,392 K []{} Perl> open I, '<', 'test.dat'; my $s = do{ local $/; <I> }; print + mem;; 1,986,016 K []{} Perl> Terminating on signal SIGINT(2)

      So maybe it's 'a windows thing'.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
      In the absence of evidence, opinion is indistinguishable from prejudice.
      I'm not sure where this mem comes from.

      BTW. I missed/ignored that at first reading. mem() is a function defined in my REPL:

      sub mem{ `tasklist /nh /fi "PID eq $$"` =~ m[(\S+ K)$] }

      And tasklist is a system utility, which with the above parameters produces:

      C:\test>\Perl22\bin\p1 [0]{} Perl> print `tasklist /nh /fi "PID eq $$"`;; perl.exe 2524 Console 1 +9,384 K

      And without the /n{o}h{eader} switch gives:

      [0]{} Perl> print `tasklist /fi "PID eq $$"`;; Image Name PID Session Name Session# Me +m Usage ========================= ======== ================ =========== ===== +======= perl.exe 2524 Console 1 +9,428 K

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
      In the absence of evidence, opinion is indistinguishable from prejudice.
      Please try
      sub psMem { my( $pid ) = @_; my @lines = qx{$GOTPS -o vsize,rss -p $pid }; chomp @lines; foreach my $line (@lines) { my ( $vsize, $rss) = split /\s+/, $line; return "VM: $vsize RSS: $rss "; } return; }
        I don't know what $GOTPS is. But I've run
        #!/usr/bin/perl use warnings; use strict; use feature qw{ say }; say $$; sleep 10; my $file = shift; open my $FH, '<', $file or die $!; my $string; { local $/; $string = <$FH> }; say length $string; sleep 1; undef $string; seek $FH, 0, 0; $string = do { local $/; <$FH> }; say length $string;

        I've used the PID reported by the script in another terminal and ran:

        while : ; do ps -o vsize,rss -p $PID | grep '[0-9]' ; done

        And got the following output:

        ($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord }map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1172023]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others examining the Monastery: (4)
As of 2024-03-29 09:17 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found