Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery

How can I let Perl use more than 3GB of RAM??

by maya_the_be (Initiate)
on Jan 22, 2008 at 15:52 UTC ( #663608=perlquestion: print w/replies, xml ) Need Help??

maya_the_be has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I have to use huge amounts of data repeatedly in a process. But when the Perl process goes above 3GB in RAM, it aborts with the "Out of memory" error. I'm running Ubuntu 32 bit server version so I have 4GB RAM enabled. The Max Memory Size is unlimited according to ulimit. How can I let Perl use more than 3GB of RAM??

yours truly, maya_the_be

#~~~~~~~~~~~~~~~~~~~~~~ ``><((@ ~~~~~
  • Comment on How can I let Perl use more than 3GB of RAM??

Replies are listed 'Best First'.
Re: How can I let Perl use more than 3GB of RAM??
by cdarke (Prior) on Jan 22, 2008 at 16:11 UTC
    In a nutshell, use a 64-bit operating system (and machine). Operating systems like Windows and Linux do not allow direct access to RAM (unless you are part of the kernel, or a device driver). Instead they use Virtual Memory.

    The program needs to access the memory area using an address, and the largest that can be held in a 32-bit word is 4GB.
    So where did the other GB go?
    That is reserved for kernel.
    Kernel uses a Gig?
    No, that address range is reserved for kernel use, it doesn't use anything like that.

    A 64-bit architecture will, in theory allow an address space of up to 16 Exabytes.
Re: How can I let Perl use more than 3GB of RAM??
by maya_the_be (Initiate) on Jan 22, 2008 at 16:47 UTC
    Hmm. If my Perl is running at 3GB, Kernel reserves 1GB, what does this mean for all the other processes, such as Nautilus and System Monitor, which are also running?

      It means they are also limited to 3 gigs maximum. The split is not 1-gig for the kernel and 3-gigs for processes, it is 1-gig for the kernel and 3-gigs PER process.

      The problem is that a 32-bit address space only gives you 4 gigs of addressable memory, so the processor has an "interpreter" called the Translation Lookaside Buffer which maps virtual addresses into physical addresses. So your perl application sees the system as having 4 gigs of memory, 1 of which is for the kernel and 3 gigs for itself. Other processes see the same thing, but the 3 gigs they are able to address may not be the same 3 gigs of physical memory that other processes see. The upside to this arrangment is that it lets you use more than 4 gigs of RAM in a 32-bit machine. The downside is that the TLB must be flushed when a different process is swapped onto the processor because the addresses that were valid for one process may not be the same addresses for another process.

      If memory is the only problem, besides the option to go to 64-bit hardware you could also look at getting (or building) a 4-gig/4-gig split kernel. Red Hat calls this kernel the 'hugemem' kernel, other distros probably have something similar. Using this kernel the kernel gets 4 gigs of address space and processes get 4 gigs each. The downside to this is that the TLB can still only deal with 4 gigs of address space, so with the 3/1 split kernel you have to flush the TLB every time you switch processes, with the 4/4 split kernel you also have to flush the TLB every time you switch from kernel mode to user mode or vice-versa. How much of an impact this has on performance varies (mostly depending on how much I/O your application does), but taking a 60% performance hit for using a 4/4 split kernel is not uncommon.

      We're not surrounded, we're in a target-rich environment!

        Of course the other option is to split the workload, if you have 18 gigs of RAM available and the application is able to split the work up among more than one process, then you can have 6 processes using 3 gigs of RAM each and use up all of that 18 gigs of RAM for your workload.

        We're not surrounded, we're in a target-rich environment!

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://663608]
Approved by Limbic~Region
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (4)
As of 2021-06-24 14:23 GMT
Find Nodes?
    Voting Booth?
    What does the "s" stand for in "perls"? (Whence perls)

    Results (128 votes). Check out past polls.