memory usage woes

[urufu-san]

My understanding of scope must be broken. I've got a small app with several subroutines, one of which reads in very large amounts of data into a hash. The data in question is a reference to a hash, averaging 25-30k keys. The values are themselves references to hashes containing seven or eight key/values. Two of those values average about 50 bytes, and the others average about ten.

This data is used only within the scope of said sub, and *should* go away once the scope of the sub has ended. Right? Watching the process with top, I can see it slowly eat up memory while going through an iteration of the sub (it is called 3 times), but not release anything once it proceeds to the next iteration.

I know there isn't really any way to "free()" memory in perl, since it does all the garbage collection, but shouldn't it release the now unused memory once the sub's scope has ended? With only three loops, this thing chews up almost 300MB of memory.

I'm using 'use vars ...' and 'use strict' to ensure that everything is correctly scoped. There are only two variables in global scope, one a short scalar and the other a three element array. Everything else is scoped within its given sub (via my). Is there anything else I can do?

[lhoward]

When a variable goes out of scope, perl frees up memory for use by other purposes within your program, but perl never releases memory to the OS (not until your program ends).

[urufu-san]

I feared as much. Maybe I'll write this thing in C after all..

[tye]

C doesn't return freed virtual memory back to the operating system either. Unless you are running out of swap space, this shouldn't be of much concern, however.

Now, Perl certainly uses more memory than most things. But avoiding Perl because some number that "top" reports that you don't even understand very well might not be the best application of your time. (:

The two most common measure of memory use are roughly "virtual memory size" and "working set size". The first measures (roughly) how much swap space a process requires while it exists. The second measures (roughly) how much physical memory the process requires to actually get work done.

These days, if you are worried about running out of swap space, then I strongly suggest you configure more swap space for your system.

If you are worried about your system running slow (because it wants more physical memory), then you need to look at the working set size, and that size will go down when the process stops using the memory that it still has allocated to its virtual address space (whether the program was written in C or in Perl).

If you want a process to return memory to the operating system, then you pretty much have to restart the program. You can do that easily in Perl with: exec( $^X, $0, @ARGV );

        - tye (but my friends call me "Tye")

[perrin]

The program should reuse memory allocated for lexical variables, but it will grow to the maximum size needed at any one time. That means that if one of the times you call this sub it uses 300MB, that 300MB will stay allocated to this perl process, but can be reused by the same lexicals if you execute the sub again.

If it seems to be adding each time, rather than reusing previously allocated memory, you may have a scoping bug in your code. It's possible that you missed a place where you are keeping a reference to some large data structure and thus preventing it from being garbage collected.