http://www.perlmonks.org?node_id=1062823


in reply to Perl Program - Out of memory?

A few things superficially come to mind:

Pursuing the last thought, I frankly do suspect that a lot of this “nested loop” logic could, indeed be expressed as a query which might, indeed, produce many thousands of rows as a “cross-product” between several smaller constituent tables.   (“And so what... that’s what SQL servers do for a living...”)   But this might then serve to rather-drastically reduce the complexity and memory-footprint of your code, which now only has to consume a record-set that is presented to it.

Note also that “SQL” doesn’t have to imply “a server.”   The SQLite database system, for instance, is built on single-files, and it runs quite nicely on everything from mainframes to cell-phones.   My essential notion here is that maybe you can shove “all that data” out of (virtual...) memory, and into file(s).