Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: array size

by Marshall (Canon)
on Aug 10, 2012 at 16:51 UTC ( [id://986773]=note: print w/replies, xml ) Need Help??


in reply to array size

If you are running 32 bit Perl like myself on Windows, then 2 GB would be an absolute "theoretical" max size due to addressing limitations and you won't be able to get all of that for array data storage. My Win XP Pro system will "auger into the ground" in terms of performance well short of that (memory swapping, etc).

Mileage varies a LOT and there are many variables involved. My largest memory hog takes about 300 MB+ of memory, but is compute and I/O bound - no significant memory swapping takes place - if that starts, you are "dead" - performance will plummet by orders of magnitude!

You should realize that a Perl data structures have significant amounts of "bookkeeping".

In practical terms, 100K things - absolutely no problem, 1 million also no problem, after that serious thinking should start about "why do I need multiple million things in memory at the same time?", especially a simple array structure which is hard to search.

Time to think about other algorithms possibly including databases. I have become quite enamored with SQLite - this works extremely well with Perl. If you are thinking about running a forking server on Unix, then then you shouldn't be thinking about even getting close to a single process limit. Run "mean and clean."

Summary: Memory usage still does matter. If you are on your own dedicated work station, that's one thing - if you want to run on a server, think even much more carefully about the algorithms and methods.

If you could describe your problem that you figure requires "max memory", maybe the Monks can help you reformulate this. At this point, I have no idea. But there are very real practical limits.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://986773]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others about the Monastery: (7)
As of 2024-03-19 11:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found