Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Perl memory limit - Windows64 strawberry perl

by sectokia (Scribe)
on Mar 18, 2021 at 00:17 UTC ( #11129866=perlquestion: print w/replies, xml ) Need Help??

sectokia has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I am using 'strawberry-perl-5.32.1.1-64bit.msi' on Windows 64bit (tried Windows10 and Server2012) to process some data files, which requires some very large hashes.

On all system I have tried perl throws "Out of memory!" to console and dies once perl.exe hits about 2.6 to 2.8GB of memory. It seems to run fine if I limit the input slightly to prevent it reaching beyond about 2.4GB

Perl -v says 'This is perl 5, version 32, subversion 1 (v5.32.1) built for MSWin32-x64-multi-thread'

Is perl somehow restricted to 32bit virtual memory limits? I was hoping to be able to go to at least 8GB.

Any ideas?

Thanks!
  • Comment on Perl memory limit - Windows64 strawberry perl

Replies are listed 'Best First'.
Re: Perl memory limit - Windows64 strawberry perl
by swl (Vicar) on Mar 18, 2021 at 01:53 UTC

    FWIW, I've run programs with 64 bit Strawberry perl that used 16+GB of RAM. That was on a machine with 64GB.

    How much RAM does your machine have? And how much was in use by other processes when you ran your checks?

    Edit 2021-03-21 - fix typo

Re: Perl memory limit - Windows64 strawberry perl
by holli (Abbot) on Mar 18, 2021 at 00:26 UTC
Re: Perl memory limit - Windows64 strawberry perl
by Discipulus (Abbot) on Mar 18, 2021 at 06:57 UTC
    Hello sectokia,

    are you sure your program is running 64bit? Check the process in the taskmanager.

    Infact if anything in the chain is 32bit (the prompt used to launch the program, dependencies, external programs etc...) the resulting program will be 32bit.

    L*

    There are no rules, there are no thumbs..
    Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.

      This was indeed the problem! A bad ENV variable from some dev software was causing windows to run a 32bit cmd.exe.

      I've learned that windows has 32bit and 64bit cmd.exe in various places (System32, wow64, amd64_microsoft_... wow64_microsoft_

      Getting back to default and 64bit perl now is able to pass 8GB ram with no issues

        Happy this helped ;)

        as you are lurking the dangerous paths of this UnfortunateOS, take a read of articles in my bibliotheca about redirection because in such OS when you suspect the OS being guilty you are probably right.

        L*

        There are no rules, there are no thumbs..
        Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
Re: Perl memory limit - Windows64 strawberry perl
by Marshall (Canon) on Mar 20, 2021 at 01:26 UTC
    I guess I am dubious about "process some data files, which requires some very large hashes".

    A "simple" DB like SQLite probably suits your purpose better than a huge in-memory hash table.
    SQLite is actually very sophisticated in how it works.

    If you could explain the application in more detail, I am sure that a less memory intensive, and higher performance implementation can be provided.

      A "simple" DB like SQLite probably suits your purpose better than a huge in-memory hash table
      Maybe. We need to learn more about the OP's requirements and algorithms. If the OP's algorithm really does need to perform vast numbers of table lookups, Perl hashes will probably be at least ten times faster than SQLite, as indicated by:

      Note that in High Performance Game of Life (and related Fastest way to lookup a point in a set) profiling indicated that the millions of hash lookups required by the simple GOL algorithm was the dominant performance hot spot ... and the overhead of calling out to (external) SQLite for these was an order of magnitude slower than native Perl hash lookups (so long as you have enough physical memory to comfortably hold the hashes).

      Buk comment: "however fast the C/C++ DB code is, calling into it from Perl adds a layer of unavoidable overhead that Perl's built-in hashes do not have"

        I think that we need to learn more about the OP's application.
        I said "probably", not for sure.
        If the hash table won't fit into memory, then we have to do something else.
Re: Perl memory limit - Windows64 strawberry perl
by eyepopslikeamosquito (Bishop) on Mar 20, 2021 at 22:51 UTC

    sectokia: To get further help on this, I think you'll need to post a Short, Self-Contained, Correct Example demonstrating the problem you are seeing in your environment. That way, other monks can run your SSCCE in their 64-bit Strawberry Perl environment and report back their findings.

    Re the OP's requirements/algorithms (e.g. using a Perl hash vs SQLite say) I just stumbled upon this recent node by the OP:

    Update: For the clearest, most repeatable test, run your test program immediately after rebooting Windows (search for reboot in The 10**21 Problem (Part 3) for one (non-Perl) example I remember where rebooting made a difference). BTW, like others in this thread, I have successfully addressed over 16 GB of memory with 64-bit Strawberry Perl on a PC with 32 GB of physical memory.

Re: Perl memory limit - Windows64 strawberry perl
by Anonymous Monk on Mar 19, 2021 at 22:18 UTC
    Maybe you need to re-consider your fundamental algorithms. When you start to delve into these "virtual" memory sizes, you start to realize that "virtual" memory is really a disk file. And, as "hash" data structures strive to randomize their memory-address references, as they are designed to do ... they run into an argument with the operating system that they are doomed to lose.
      Sounds (on the surface) like he knows what he's talking about; adds zero value to the discourse.
A reply falls below the community's threshold of quality. You may see it by logging in.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11129866]
Approved by GrandFather
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (4)
As of 2021-10-19 18:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My first memorable Perl project was:







    Results (77 votes). Check out past polls.

    Notices?