One object per line, some of which (average of 20 to 50 depending on the data) are subobjects of other objects. So I have like 50000 busses and about 33 objects with timetable information each. | [reply] |
Ouch! It's not your data, but your objects that are killing you.
I just loaded 2_000_000 objects ( hash-based, each containing 1 key - scalar 8 bytes. 1 value - string 80 bytes). I put the 2_000_000 object handles into a hash keyed by the id.
Total data 2M * 80 = 152 MB.
Total memory consumed 700MB.
Add a few nested objects in each of those objects and you can see where your memory is going to.
Imagine if perl stored unicode as objects, 1 character per, and strings were arrays of theose objects. A convenient OO-schema, but with today's hardware and today's memory limits, totally impractical.
There has to be a better way.
Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
"Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon
| [reply] |
