in reply to XML::CSV out of memory

In all honesty if you have a modern amount of RAM on your system, 500MB isn't that big.

That said, in the docs, it looks like you can do this:

use XML::CSV;

$default_obj_xs = Text::CSV_XS->new({quote_char => '"'});

$csv_obj = XML::CSV->new({csv_xs => $default_obj_xs}); 

$csv_obj->{column_headings} = \@arr_of_headings;

$csv_obj->{column_data} = \@arr_of_data;

$csv_obj->print_xml("out.xml");

[download]

Basically, notice that you can pass a Text::CSV_XS object to XML::CSV, which would allow you to read in $n lines of the CSV file, and then pass them in as an array to XML::CSV. This would be fairly trivial to do, but if you can't get it figured out, come back and we can help you through it.

Three thousand years of beautiful tradition, from Moses to Sandy Koufax, you're god damn right I'm living in the fucking past

Replies are listed 'Best First'.
Re^2: XML::CSV out of memory
by monkey_boy (Priest) on Mar 26, 2015 at 16:25 UTC
    I guess we dont know how much RAM is required without seeing the input CSV data, and the derived XML, the in memory parse of the CSV will be many factors larger than the original file.
    That said, if you have access to a machine with a lot more RAM this can't hurt.


    This is not a Signature...
[reply]

      I agree but, even the smallest linode and EC2 instances have a gig of RAM, and DigitalOcean is 512MB, which squeaks by. I digress, but I'm just saying, it would be a big difference it it were a 500GB file being parsed.

      Three thousand years of beautiful tradition, from Moses to Sandy Koufax, you're god damn right I'm living in the fucking past

[reply]
Re^2: XML::CSV out of memory
by slugger415 (Monk) on Mar 26, 2015 at 16:58 UTC

    Thanks for the replies and suggestions. Yes it is strange, the system now has 12 gigs of memory. My own workstation has 16 gigs and I'm not getting the out of memory error running the same script. Anyway.

    So it's probably a dumb question, but do I even need Text::CSV for your suggestion? Couldn't the script just put $n lines of text into an array and feed those to @arr_of_data? And how would I do $csv_obj->print_xml("out.xml") at each iteration without clobbering the previous iteration? (ok, you can tell I'm still learning here - begging patience...)

    Thanks.

[reply]
      do I even need Text::CSV for your suggestion? Couldn't the script just put $n lines of text into an array and feed those to @arr_of_data?

      Text::CSV will insure you are reading complete records. If you can be sure that your input files have no embedded "new lines" in the CSV records, then you could skip Text::CSV.

      And how would I do $csv_obj->print_xml("out.xml") at each iteration without clobbering the previous iteration?

      Try (untested):

      open(my $outFH, '>', "out.xml");
while (<>) {
    ...;
    $csv_obj->print_xml($outFH);
}

      [download]
[reply]
[d/l]

      The example seems to call for Text::CSV_XS in this instance:

      $default_obj_xs = Text::CSV_XS->new({quote_char => '"'});
$csv_obj = XML::CSV->new({csv_xs => $default_obj_xs});

      [download]

      It's really not hurting you to use it, so you might as well honestly.

      Three thousand years of beautiful tradition, from Moses to Sandy Koufax, you're god damn right I'm living in the fucking past

[reply]
[d/l]

In Section Seekers of Perl Wisdom