Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

memory usage woes

by urufu-san (Novice)
on Nov 20, 2001 at 23:32 UTC ( #126591=perlquestion: print w/replies, xml ) Need Help??
urufu-san has asked for the wisdom of the Perl Monks concerning the following question:

My understanding of scope must be broken. I've got a small app with several subroutines, one of which reads in very large amounts of data into a hash. The data in question is a reference to a hash, averaging 25-30k keys. The values are themselves references to hashes containing seven or eight key/values. Two of those values average about 50 bytes, and the others average about ten.

This data is used only within the scope of said sub, and *should* go away once the scope of the sub has ended. Right? Watching the process with top, I can see it slowly eat up memory while going through an iteration of the sub (it is called 3 times), but not release anything once it proceeds to the next iteration.

I know there isn't really any way to "free()" memory in perl, since it does all the garbage collection, but shouldn't it release the now unused memory once the sub's scope has ended? With only three loops, this thing chews up almost 300MB of memory.

I'm using 'use vars ...' and 'use strict' to ensure that everything is correctly scoped. There are only two variables in global scope, one a short scalar and the other a three element array. Everything else is scoped within its given sub (via my). Is there anything else I can do?

Replies are listed 'Best First'.
Re: memory usage woes
by lhoward (Vicar) on Nov 20, 2001 at 23:47 UTC
    When a variable goes out of scope, perl frees up memory for use by other purposes within your program, but perl never releases memory to the OS (not until your program ends).
      I feared as much. Maybe I'll write this thing in C after all..

        C doesn't return freed virtual memory back to the operating system either. Unless you are running out of swap space, this shouldn't be of much concern, however.

        Now, Perl certainly uses more memory than most things. But avoiding Perl because some number that "top" reports that you don't even understand very well might not be the best application of your time. (:

        The two most common measure of memory use are roughly "virtual memory size" and "working set size". The first measures (roughly) how much swap space a process requires while it exists. The second measures (roughly) how much physical memory the process requires to actually get work done.

        These days, if you are worried about running out of swap space, then I strongly suggest you configure more swap space for your system.

        If you are worried about your system running slow (because it wants more physical memory), then you need to look at the working set size, and that size will go down when the process stops using the memory that it still has allocated to its virtual address space (whether the program was written in C or in Perl).

        If you want a process to return memory to the operating system, then you pretty much have to restart the program. You can do that easily in Perl with: exec( $^X, $0, @ARGV );

                - tye (but my friends call me "Tye")
Re: memory usage woes
by perrin (Chancellor) on Nov 20, 2001 at 23:51 UTC
    The program should reuse memory allocated for lexical variables, but it will grow to the maximum size needed at any one time. That means that if one of the times you call this sub it uses 300MB, that 300MB will stay allocated to this perl process, but can be reused by the same lexicals if you execute the sub again.

    If it seems to be adding each time, rather than reusing previously allocated memory, you may have a scoping bug in your code. It's possible that you missed a place where you are keeping a reference to some large data structure and thus preventing it from being garbage collected.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://126591]
Approved by root
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (3)
As of 2017-08-19 10:08 GMT
Find Nodes?
    Voting Booth?
    Who is your favorite scientist and why?

    Results (310 votes). Check out past polls.