Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Re: tuning hash memory usage

by gaspodethewonderdog (Monk)
on Nov 20, 2000 at 23:36 UTC ( #42569=note: print w/replies, xml ) Need Help??


in reply to tuning hash memory usage

Having dealt with sorting, rebuilding and other sorts of fun on data sets that were well into the 300+MB amounts of data and only having 128MB or so of memory I can give you a few pointers/suggestions.

1) Expect Perl to use much more memory to store your data than what you expect it to be. I have found that it is generally safe to assume that if you have 50MB of data expect it the program to require from 70MB to 100MB. Perl favors using more memory than CPU cycles to solve a problem.

2) For any significantly large amounts of information it is best to use multiple arrays over hashes (or worse hashes of hashes) or objects. Unless you have the memory, you won't be able to store it all in memory and you'll kill your performance swapping to memory.

3) Remove all unneeded data. If all you need is a list of search/replace information and file names, but you also have dates, file sizes and other 'useless' information then get rid of it and then write functions to get that information from what you are storing in memory.

Unless you have a specific reason for using objects to do this work I'd suggest trying a more basic solution. There are certainly tons of cool things you can do with objects, but if you don't have the system resources (memory) to work with them, then you either have to try something else. I suppose some form of caching might help and you might even be able to let the OS do this by making the swap file big enough, but if you need to check a lot of objects quickly I'm not certain this would give you a big win.

Best of luck!

Replies are listed 'Best First'.
Re: Re: tuning hash memory usage
by jmason (Novice) on Nov 21, 2000 at 23:45 UTC
    3) Remove all unneeded data. If all you need is a list of search/replace information and file names, but you also have dates, file sizes and other 'useless' information then get rid of it and then write functions to get that information from what you are storing in memory.

    Yes, a good tip. I've found that changing some flags (is_this_type_of_content, is_that_type etc.) from setting their default value explicitly, so instead undef is used as the default value, saves a lot of memory -- as the key and value just don't have to be set at all for this to work.

    I think I'll have to serialize the dependencies array into something much smaller, probably using (at the easy end) a serialize-into-string trick and (at the heavier end) some kind of Flyweight-pattern-style trickery.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://42569]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (3)
As of 2023-02-04 02:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    I prefer not to run the latest version of Perl because:







    Results (30 votes). Check out past polls.

    Notices?