|
Re: Perl memory limit - Windows64 strawberry perl
by swl (Vicar) on Mar 18, 2021 at 01:53 UTC
|
FWIW, I've run programs with 64 bit Strawberry perl that used 16+GB of RAM. That was on a machine with 64GB.
How much RAM does your machine have? And how much was in use by other processes when you ran your checks?
Edit 2021-03-21 - fix typo
| [reply] |
Re: Perl memory limit - Windows64 strawberry perl
by holli (Abbot) on Mar 18, 2021 at 00:26 UTC
|
| [reply] |
Re: Perl memory limit - Windows64 strawberry perl
by Discipulus (Canon) on Mar 18, 2021 at 06:57 UTC
|
Hello sectokia,
are you sure your program is running 64bit? Check the process in the taskmanager.
Infact if anything in the chain is 32bit (the prompt used to launch the program, dependencies, external programs etc...) the resulting program will be 32bit.
L*
There are no rules, there are no thumbs..
Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
| [reply]
[d/l] |
|
|
This was indeed the problem! A bad ENV variable from some dev software was causing windows to run a 32bit cmd.exe.
I've learned that windows has 32bit and 64bit cmd.exe in various places (System32, wow64, amd64_microsoft_... wow64_microsoft_
Getting back to default and 64bit perl now is able to pass 8GB ram with no issues
| [reply] |
|
|
| [reply]
[d/l]
[select] |
Re: Perl memory limit - Windows64 strawberry perl
by Marshall (Canon) on Mar 20, 2021 at 01:26 UTC
|
I guess I am dubious about "process some data files, which requires some very large hashes".
A "simple" DB like SQLite probably suits your purpose better than a huge in-memory hash table.
SQLite is actually very sophisticated in how it works.
If you could explain the application in more detail, I am sure that a less memory intensive, and
higher performance implementation can be provided.
| [reply] |
|
|
A "simple" DB like SQLite probably suits your purpose better than a huge in-memory hash table
Maybe. We need to learn more about the OP's requirements and algorithms.
If the OP's algorithm really does need to perform vast numbers of table lookups, Perl hashes will probably be at least ten times
faster than SQLite, as indicated by:
Note that in High Performance Game of Life (and related Fastest way to lookup a point in a set) profiling indicated that the millions of hash lookups required by the simple GOL algorithm was the dominant performance hot spot ... and the overhead of calling out to (external) SQLite for these was an order of magnitude slower than native Perl hash lookups (so long as you have enough physical memory to comfortably hold the hashes).
Buk comment: "however fast the C/C++ DB code is, calling into it from Perl adds a layer of unavoidable overhead that Perl's built-in hashes do not have"
| [reply] |
|
|
I think that we need to learn more about the OP's application.
I said "probably", not for sure.
If the hash table won't fit into memory, then we have to do something else.
| [reply] |
|
|
Re: Perl memory limit - Windows64 strawberry perl
by eyepopslikeamosquito (Bishop) on Mar 20, 2021 at 22:51 UTC
|
sectokia: To get further help on this, I think you'll need to post a Short, Self-Contained, Correct Example
demonstrating the problem you are seeing in your environment.
That way, other monks can run your SSCCE in their
64-bit Strawberry Perl environment and report back their findings.
Re the OP's requirements/algorithms (e.g. using a Perl hash vs SQLite say) I just stumbled upon this recent node by the OP:
Update: For the clearest, most repeatable test, run your test program immediately after rebooting Windows (search for reboot in The 10**21 Problem (Part 3) for one (non-Perl) example I remember where rebooting made a difference). BTW, like others in this thread, I have successfully addressed over 16 GB of memory with 64-bit Strawberry Perl on a PC with 32 GB of physical memory.
| [reply] |
Re: Perl memory limit - Windows64 strawberry perl
by Anonymous Monk on Mar 19, 2021 at 22:18 UTC
|
Maybe you need to re-consider your fundamental algorithms. When you start to delve into these "virtual" memory sizes, you start to realize that "virtual" memory is really a disk file. And, as "hash" data structures strive to randomize their memory-address references, as they are designed to do ... they run into an argument with the operating system that they are doomed to lose. | [reply] |
|
|
Sounds (on the surface) like he knows what he's talking about; adds zero value to the discourse.
| [reply] |
| A reply falls below the community's threshold of quality. You may see it by logging in. |