Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery

Re^2: Perl memory limit - Windows64 strawberry perl

by eyepopslikeamosquito (Bishop)
on Mar 20, 2021 at 06:52 UTC ( #11129998=note: print w/replies, xml ) Need Help??

in reply to Re: Perl memory limit - Windows64 strawberry perl
in thread Perl memory limit - Windows64 strawberry perl

A "simple" DB like SQLite probably suits your purpose better than a huge in-memory hash table
Maybe. We need to learn more about the OP's requirements and algorithms. If the OP's algorithm really does need to perform vast numbers of table lookups, Perl hashes will probably be at least ten times faster than SQLite, as indicated by:

Note that in High Performance Game of Life (and related Fastest way to lookup a point in a set) profiling indicated that the millions of hash lookups required by the simple GOL algorithm was the dominant performance hot spot ... and the overhead of calling out to (external) SQLite for these was an order of magnitude slower than native Perl hash lookups (so long as you have enough physical memory to comfortably hold the hashes).

Buk comment: "however fast the C/C++ DB code is, calling into it from Perl adds a layer of unavoidable overhead that Perl's built-in hashes do not have"

  • Comment on Re^2: Perl memory limit - Windows64 strawberry perl

Replies are listed 'Best First'.
Re^3: Perl memory limit - Windows64 strawberry perl
by Marshall (Canon) on Mar 21, 2021 at 23:34 UTC
    I think that we need to learn more about the OP's application.
    I said "probably", not for sure.
    If the hash table won't fit into memory, then we have to do something else.

      Nowadays, many home PCs (including mine) have 32 GB of memory and a 64-bit OS ...

      -- from The 10**21 Problem (Part 2) (2014)

      Agreed. Given the OP was "hoping to be able to go to at least 8GB" -- and given I had 32 GB of memory on my home PC way back in 2014 (and I am definitely not a rich man :) -- the simplest and cheapest something else may be simply "buy more memory". Well, that seems attractive for a commercial company because the cost of PC hardware is dwarfed by the cost of people (and the cost/risk associated with making massive code changes to critical production systems).

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11129998]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (8)
As of 2023-02-05 14:30 GMT
Find Nodes?
    Voting Booth?
    I prefer not to run the latest version of Perl because:

    Results (31 votes). Check out past polls.