Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re: Text CSV_XS memory crash

by Anonyrnous Monk (Hermit)
on Feb 02, 2011 at 15:07 UTC ( #885755=note: print w/ replies, xml ) Need Help??


in reply to Text CSV_XS memory crash

{push @array, $row;}

I'd suppose you're simply running out of memory, because you're collecting all the data in the @array.  2 GB memory usage for representing 100 MB file contents in an array (of arrays) structure isn't that unusual.

Do you actually need to hold the entire data in memory, or might there perhaps be a way to process things sequentially?


Comment on Re: Text CSV_XS memory crash
Select or Download Code
Re^2: Text CSV_XS memory crash
by glepore70 (Novice) on Feb 02, 2011 at 15:24 UTC
    Since I'm acting on the column values, I think I have to read every line into the array, in order to pull out the first value of every row to get a column. Perhaps I'm missing a CSV_XS method for selecting the column instead of reading in every row?
      ...in order to pull out the first value of every row to get a column.

      Not sure I'm understanding you correctly, but if you only need the first column of every row, why not store only the first column (that would at least reduce memory usage somewhat).

      The getline() method returns a reference to an array holding the columns. In other words, $row->[0] would be the first column.

      If, OTOH, you actually do need access to all columns of all rows simultaneously, I'm afraid there's not much you can do except to upgrade memory (or write out the data into another (DB) file format that allows direct random access to individual fields).

        Hmm, I'm starting to see the light. I only need to work on one column at a time, read in the column, select min/max and distinct, then load the next column and repeat.

        I'll go back to the drawing board, I think I'm pretty close...

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://885755]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (13)
As of 2014-07-28 13:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (200 votes), past polls