Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re^3: Text CSV_XS memory crash

by Anonyrnous Monk (Hermit)
on Feb 02, 2011 at 15:41 UTC ( [id://885770]=note: print w/replies, xml ) Need Help??


in reply to Re^2: Text CSV_XS memory crash
in thread Text CSV_XS memory crash

...in order to pull out the first value of every row to get a column.

Not sure I'm understanding you correctly, but if you only need the first column of every row, why not store only the first column (that would at least reduce memory usage somewhat).

The getline() method returns a reference to an array holding the columns. In other words, $row->[0] would be the first column.

If, OTOH, you actually do need access to all columns of all rows simultaneously, I'm afraid there's not much you can do except to upgrade memory (or write out the data into another (DB) file format that allows direct random access to individual fields).

Replies are listed 'Best First'.
Re^4: Text CSV_XS memory crash
by glepore70 (Novice) on Feb 02, 2011 at 15:57 UTC
    Hmm, I'm starting to see the light. I only need to work on one column at a time, read in the column, select min/max and distinct, then load the next column and repeat.

    I'll go back to the drawing board, I think I'm pretty close...

      select min/max and distinct

      FWIW, things like min/max (and many other statistics) can also be computed sequentially (or incrementally), i.e. without storing all values in memory prior to starting the computations.  Maybe that's worth reconsidering...

      Have you considered using DBD::CSV, or alternativly importing your data into a database of some sort?

        Using DBD::CSV will never be a valid option when reading a CSV file in full will cause memory hogs, as DBD::CSV will read the whole file into memory always. Nothing to prevent that. That means that the memory consumption for CSV files accessed with DBI might be up to double the size of the complete dataset as returned bi Text::CSV_XS->getline_all ().


        Enjoy, Have FUN! H.Merijn

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://885770]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others romping around the Monastery: (3)
As of 2025-02-17 01:16 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found