I have a programm, that requires text files to be read into memory. The size of these files started as "small" 5 megabyte ones but has increased to about 125 megabytes. Loading these files into objects of my class structure requires about 1.5 GB of memory.
I cannot read only parts of this file as the data needs to be sorted in a very complicated way.
I now try to reduce the memory required. Unfortunately there seems to be little information about how to do so. I tried a couple of things so far:
- The text file I read, has fixed columns sizes, instead of splitting these into my object hash's key-value pairs, I just stored the lines into my objects using the substr function for all get-methods. I did that because I read that a scalar, requires a minimum amount of memory that would exceed the length of most of my values size. The effect was an obviously increased runtime as substr now was called about 16 million times per object, but no remarkable reduction in memory usage, which made me conclude that the minimum size required for a hash's key-value pair is much closer to zero than of ?32? Bytes for a scalar.
I also tried limiting memory usage by removing useless whitespaces within a value, but there are few and memory reduction was less than 2 percent.
My last and more successful effort was to reduce memory by leaving all values that have a standard value as undefined and just return the standard value. This reduced memory by about ten percent but still that is much less than required.
Are there any rescources about reducing memory usage out there? Anything that makes me understand how memory is allocated in scalars/arrays/hashes? Or can I possibly reduce the usage by enconding my values into strings using a wider range of characters and reducing the length of a string but still enabling me to sort those elements without decoding the values as this would very likely take too much time again.