Re: tuning hash memory usage

[gaspodethewonderdog]

Having dealt with sorting, rebuilding and other sorts of fun on data sets that were well into the 300+MB amounts of data and only having 128MB or so of memory I can give you a few pointers/suggestions.

1) Expect Perl to use much more memory to store your data than what you expect it to be. I have found that it is generally safe to assume that if you have 50MB of data expect it the program to require from 70MB to 100MB. Perl favors using more memory than CPU cycles to solve a problem.

2) For any significantly large amounts of information it is best to use multiple arrays over hashes (or worse hashes of hashes) or objects. Unless you have the memory, you won't be able to store it all in memory and you'll kill your performance swapping to memory.

3) Remove all unneeded data. If all you need is a list of search/replace information and file names, but you also have dates, file sizes and other 'useless' information then get rid of it and then write functions to get that information from what you are storing in memory.

Unless you have a specific reason for using objects to do this work I'd suggest trying a more basic solution. There are certainly tons of cool things you can do with objects, but if you don't have the system resources (memory) to work with them, then you either have to try something else. I suppose some form of caching might help and you might even be able to let the OS do this by making the swap file big enough, but if you need to check a lot of objects quickly I'm not certain this would give you a big win.

Best of luck!

[jmason]

3) Remove all unneeded data. If all you need is a list of search/replace information and file names, but you also have dates, file sizes and other 'useless' information then get rid of it and then write functions to get that information from what you are storing in memory.

Yes, a good tip. I've found that changing some flags (is_this_type_of_content, is_that_type etc.) from setting their default value explicitly, so instead undef is used as the default value, saves a lot of memory -- as the key and value just don't have to be set at all for this to work.

I think I'll have to serialize the dependencies array into something much smaller, probably using (at the easy end) a serialize-into-string trick and (at the heavier end) some kind of Flyweight-pattern-style trickery.