If you data does not fit in the available memory you will have to use an algorithm that does not require random access to the data.
in reply to Parallel::Forkmanager and large hash, running out of memory
In practice that means you have to use one of the following alternatives:
- Compacting/compressing your data in some way so that it fits in the available RAM
- Sorting the data and then processing it sequentially.
- Using a multi-pass approach. The data is divided in ranges that fit on the available memory and then, you process *all* the data repeatly but only considering the data in one range every time.