Re^4: Text CSV_XS memory crash

[glepore70]

Hmm, I'm starting to see the light. I only need to work on one column at a time, read in the column, select min/max and distinct, then load the next column and repeat.

I'll go back to the drawing board, I think I'm pretty close...

[Anonyrnous Monk]

select min/max and distinct

FWIW, things like min/max (and many other statistics) can also be computed sequentially (or incrementally), i.e. without storing all values in memory prior to starting the computations.  Maybe that's worth reconsidering...

[marto]

Have you considered using DBD::CSV, or alternativly importing your data into a database of some sort?

[Tux]

Using DBD::CSV will never be a valid option when reading a CSV file in full will cause memory hogs, as DBD::CSV will read the whole file into memory always. Nothing to prevent that. That means that the memory consumption for CSV files accessed with DBI might be up to double the size of the complete dataset as returned bi Text::CSV_XS->getline_all ().