Re: Perl Memory problem ... (Memory Tools References)

[eyepopslikeamosquito]

If you give us more information we should be able to help. Meanwhile, these old nodes might give you a clue as to what your problem might be.

Using this node for Memory Tools References nowadays:

• Re^2: Perl, C and deallocation responsibility (Returning Memory back to the OS References) - Perl on Linux and Windows (unlike most other Perl platforms) does return "big" chunks of memory back to the OS

• Losing my memory
• Perl Memory problem ...
• memory leak
• Memory leak detection
• Timing of garbage collection

• Memory is not released back to operating system
• How to return unused memory to OS? (my reply)
• demonstrate that perl can give back memory to the OS
• Not able to release memory

• Out of memory during "large" request
• Memory Leak when slurping files in a loop
• Another "out of memory!" problem
• Long running tasks, perl and garbage collection
• Memory leaks and circular references
• Valgrind, Perl, and banishing leaks
• Garbage Collection and undef
• undefining hashes to free memory

General Areas to consider when faced with a Memory Leak

• Stability: Once your process grows to a certain size, does it stay there, or does it grow further over time?
• Large CPAN Modules. Are you using any?
• Circular references. See Eliminate circular reference memory leak using weaken (Perl Maven).
• Scoping.
• File/Database handling.
• Tools. Try one or more from the following section.

Perl Tools and Advice

• Profiling Memory Usage talk by Tim Bunce (YAPC::NA Austin Texas 2013 featuring Devel::SizeMe). Resident size vs vm size, rss, segments, /proc/pid/maps (e.g. SO maps question), malloc and the heap, perl internals, pads, constant folding, arenas, recursion, perl data structures (lots of pointers), ... (update: this reply describes memory used by lexical variables in subroutines at the 09:20 minute mark)
• Plans for adding memory profiling to Devel::NYTProf
• Devel::NYTProf
• Devel::Gladiator
• Devel::LeakTrace
• Test::LeakTrace
• Devel::Leak
• Devel::FindRef
• Devel::Cycle
• Test::Memory::Cycle
• Devel::MemoryTrace::Light

• EXTERNAL TOOLS FOR DEBUGGING PERL at perlhack
• FFI::Platypus

• Devel::SizeMe by Tim Bunce (mentioned in talk above)
• Devel::Size
• Devel::DumpSizes
• Devel::Arena
• Devel::Mallinfo
• Devel::Peek

• splain (diagnostics, splain - produce verbose warning diagnostics)
• perldiag (search for "out of memory", "malloc", etc)
• perlvar (search for "$^M")
• perlfaq3 (search for "malloc", "memory", ..., perl -V:usemymalloc displays value of "usemymalloc" configuration variable)
• How can I free an array or hash so my program shrinks? (perlfaq3)
• How can I make my Perl program take less memory? (perlfaq3)

• INSTALL (Build and Installation guide for perl 5, ... search for malloc)
• perl malloc.c source code

• Perl::Critic - static code analysis for Perl
• MCE::Shared

• Mini-Tutorial: Perl's Memory Management by ikegami
• As pointed out by dave_the_m: "once the perl interpreter has exhausted all memory, there's very little it can do. There is a reserved fixed buffer to allow it to display the OOM error itself, but doing anything complex like displaying a Carp::confess()-like stack backtrace would require many memory allocations to generate the display, all of which would themselves fail".

General Tools

• Memory leak (wikipedia)
• Memory debugger (wikipedia - contains a list of memory debugging tools)
• Top 20 list of memory leak detection tools (softwaretestinghelp)
• Valgrind (see also: Test::Valgrind mentioned here)
• AddressSanitizer (wikipedia)
• VTune memory profiler (wikipedia) - a performance analysis tool for x86 based machines running Linux or Microsoft Windows operating systems

• Clang (Clang is a compiler front end for the C, C++, Objective-C, and Objective-C++ programming languages ... includes a static analyzer, and several code analysis tools ... Clang operates in tandem with the LLVM compiler back end)
• LLVM (LLVM is a set of compiler and toolchain technologies that can be used to develop a front end for any programming language and a back end for any instruction set architecture ... designed around a language-independent intermediate representation (IR) that serves as a portable, high-level assembly language that can be optimized with a variety of transformations over multiple passes)
• Boehm garbage collector aka Boehm–Demers–Weiser garbage collector (wikipedia)

Static Code Analysis

• Static code analysis (wikipedia)
• List of tools for static code analysis (wikipedia)
• CodeSonar static code analysis (wikipedia)
• Coverity static code analysis (wikipedia)
• Dynamic program analysis (wikipedia)

• Duplicate code (wikipedia)
• DRY (wikipedia)
• Rule of three (wikipedia)

Code Profiling and Optimization

• Re: declaring lexical variables in shortest scope: performance? (on Code Optimization and Performance References)

Testing Concurrent Software

• Re: Threads or no Threads (Threading, Forking, Signals, Event Loop and Concurrency References)

Security Testing

• Re: Security techniques every programmer should know (Security References) (long list of Security references)

Some Related PM Nodes

• Is $^M a leftover April Fool? by BrowserUk (2003)
• How use $^M? by xiaoyafeng (2021)
• Make Perl use real malloc by creamygoodness (2011)

• Re^2: Do subroutine variables get destroyed? (Deterministic Destructor References) (discusses deterministic destructors, tracing vs reference-counted garbage collectors, RAII, ...)
• Re^2: A short whishlist of Perl5 improvements leaping to Perl7 (compares Perl's simple lexical scoping with Python's global and nonlocal keywords)

More Nodes Added Later

• Small Hash a Gateway to Large Hash? by lsherwood (2014)
• write hash to disk after memory limit by hailholyghost (2015)
• Do subroutine variables get destroyed? by bt101 (2016)
• Perl and Garbage Collection by Svetlana (2016)

• Perl "Out of memory!" Error signal backtrace by OnEhIppY (2021)
• Memory efficient design by harangzsolt33 (2022)
• Memory usage double expected by sectokia (2022)

• Re: FFI::Platypus: Replace malloc with GC_MALLOC? by ikegami (2022)
• Re^3: FFI::Platypus: Replace malloc with GC_MALLOC? by me (2022)
• Perl, C and deallocation responsibility by SheWolf (2023)
• Re^2: Perl, C and deallocation responsibility (Returning Memory back to the OS References) by me (2023)
• Re: Tk-804.036 build failure by syphilis (2023) - 32-bit build ok, 64-bit build crashes (also gcc v clang)

• Re^3: Extracting selected fields form file record (don't use File::Slurp References) by me (2023) - see haukex reply
• Does Perl have a baked-in memory llimit? by cormanaz (2023) - out of memory File::Slurp-ing a half-gig file, with 32 GB of system memory (32 v 64 bit perl build?)
• Memory limitation of a perl process in Unix by asinghvi (2004) - out of memory: is there a memory limit on a perl process on Unix/Solaris?
• Re^5: Is there something faster than string concatenation? (hermit crabs) by me (2007) - performance of Perl string concatenation depends on performance of the underlying C library realloc function

• Strawberry: Both IO and global match are VERY SLOW. Unless pre-heated (but why?) by anonymonk (2023) - see replies by syphilis, Discipulus and tonyc

See Also

• Re: perl compiler optimizer curiosity (Troubleshooting and Logging/Debugging References)
• Re: eval to replace die? (Exceptions and Error Handling References)

Updated: Many extra references were added long after the original reply was made.