Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Parsing CSV only returns the second line of the file

by saint_geser (Initiate)
on Sep 01, 2012 at 04:22 UTC ( #991118=perlquestion: print w/ replies, xml ) Need Help??
saint_geser has asked for the wisdom of the Perl Monks concerning the following question:

Hi guys, I'm trying to create a script that parses the original csv file prints out information into a temporary file and then replaces the original file with a temp one.

So original data look something like:

1,6064.86,85391.25,593.75,13.25 2,6072.17,85392.95,593.79,13.29 3,6078.94,85393.05,593.76,13.26 4,6085.51,85392.22,593.77,13.27

and so on. I need to insert two lines at the top, two lines at the bottom, switch columns 2 and 3 around and replace column 5 with column 1.I'm using Text::CSV for parsing and also Lava GUI package. So part of the code that does parsing looks like this:

my $csv = Text::CSV->new(); open(OLD, '+<', $file) or die Lava::Message("Can't open original file" +); while (<OLD>) { next if ($. == 1); if ($csv->parse($_) { my @columns = $csv->fields(); # open new file for editing open (TEMP, '>', $temp) or die Lava::Message("Can't open tem +porary file"); # extract pattern id from file name my $pattern = substr $file, -12 , 8; #date formatting doesn't work properly so i'll get the date +from user my $select_date = ' '; my $date_panel = new Lava::Panel; $date_panel->text(' '); $date_panel->text(" Script Version: $Version"); $date_panel->text(' '); $date_panel->text(" SITE: YANDI"); $date_panel->text(' '); $date_panel->item("Type the date in format DD-Mon-YY: ", \$s +elect_date, 9); $date_panel->execute('Date Selection') or return 0; #now print first two lines into the TEMP file print TEMP "$pattern ,"; print TEMP "$select_date"; print TEMP ",,Dist=Metres\n"; print TEMP "0,0.000,0.000,0.000,0.000,0.000,0.000\n"; while( <OLD> ) { print TEMP $_; last if $. % 1; } # print the parsed body of old file print TEMP "$columns[0], $columns[2], $columns[1], $columns[ +3], $columns[0]\n"; } } # insert new lines at the end print TEMP "\n0, 0.000, 0.000, 0.000,\n"; print TEMP "0, 0.000, 0.000, 0.000, END\n"; close TEMP; close OLD; copy $temp, $file; #now we delete the temporary file Lava::Show("Deleting temporary file"); unlink $temp or Lava::Message "Couldn't delete the temporary file!"; END;

So far everything works mostly fine and the i get my columns in correct order but for some reason it only prints the 2nd line of rearranged data and then moves on to insert text at the bottom. Anyone knows where I'm doing something wrong?

Comment on Parsing CSV only returns the second line of the file
Select or Download Code
Re: Parsing CSV only returns the second line of the file
by Anonymous Monk on Sep 01, 2012 at 06:48 UTC

    Anyone knows where I'm doing something wrong?

    One loop inside another, without adequate understanding of files and modulus operator

    $ perl -le " for(1..20){printf qq{%s %s\n}, $_, int($_ % 1 ); } " 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0 12 0 13 0 14 0 15 0 16 0 17 0 18 0 19 0 20 0

      Could you please elaborate on that. I don't know much about perl, you're right but it is really annoying to do all these files by hand.

        Could you please elaborate on that.

        Find places in your code, say, a loop inside a loop, where you use the modulus operator, say like "% 1"

        Then compare that line to my code, esp the output

        When do you think that inner loop will end?

        Here is a better one, what do you think the inner while loop does, what is its purpose?

        Here is another hint, how many times do you get this warning (after you add the code)?  if( warn "calling parse " and $csv->parse($_) ) {

Re: Parsing CSV only returns the second line of the file
by Athanasius (Prior) on Sep 01, 2012 at 07:24 UTC

    Hello saint_geser, and welcome to the Monastery!

    In addition to the modulus problem highlighted by Anonymous Monk, above (and ignoring the fact that the code snippet you gave doesn’t compile!), there is a problem with the following line:

    open (TEMP, '>', $temp) or die Lava::Message("Can't open temporary fil +e");

    This occurs within the outer while loop, so on each iteration, whenever the if condition succeeds, the temp file is truncated (“clobbered,” erased) as it is re-opened for output. See open.

    Move the open statement to before the loop, so it is executed only once.

    Hope that helps,

    Athanasius <°(((><contra mundum

      Thanks guys for your replies! I'm still quite confused. This is my first perl script and I haven't done any programming since uni.

      The reason it wouldn't compile (works fine for me) is because I have perl as a part of proprietary mining software which has their own extension modules installed, e.g. Lava. I don't get any errors or warnings when running it.

      The part of code that does the parsing I got from somewhere on the internet and modified for my purpose and I'm not quite sure where the inner while cycle is from. I'm pretty sure it shouldn't be there.

      So I've commented out the inner cycle and moved the part where I open TEMP file for editing and insert two lines up the top out of the outer cycle. Now this part of code looks like this:

      while ( <OLD> ) { next if ($. == 1); if ($csv->parse($_)) { my @columns = $csv->fields(); # print the parsed body of old file print TEMP "$columns[0], $columns[2], $columns +[1], $columns[3], $columns[0]\n"; #while( <OLD> ) #{ #print TEMP $_; #last if $. % 1; #} } else { my $err = $csv->error_input; Lava::Message("Failed to parse line: $err"); } }

      The output that I get now has only the top 2 and bottom 2 lines:

      W1582165 ,01-Sep-12,,Dist=Metres 0, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000 0, 0.000, 0.000, 0.000, 0, 0.000, 0.000, 0.000, END

      I still can't figure out how to make parsing work properly. I'm actually very confused about this line:

      next if ($. == 1);

      It says next if line number equals to 2? In the example that I saw it parsed the whole file but here does it just parse the second line?

        The reason it wouldn't compile (works fine for me) is because I have perl as a part of proprietary mining software which has their own extension modules installed, e.g. Lava. I don't get any errors or warnings when running it.

        Then what you posted isn’t exactly what you’re running, since this line in the original post (fixed in the new post, I see):

        if ($csv->parse($_)

        is missing a closing parenthesis, and the final line:

        END;

        should be __END__.

        It says next if line number equals to 2?

        No, array elements start counting at zero, but line numbers begin at one. So that statement says: Skip the first line of data. (Presumably, the original code expected the first line to be a heading?)

        You are making progress, but it’s difficult to say why your code is failing without a complete, self-contained script. Also, detailing the output you expect/desire would help the monks know what you are trying to achieve. Please see How do I post a question effectively?.

        Update: Minor edit.

        Athanasius <°(((><contra mundum

Re: Parsing CSV only returns the second line of the file
by philiprbrenan (Monk) on Sep 01, 2012 at 11:33 UTC

    Please consider building and manipulating the entire data structure in an array step by step, it makes it easier because you can see what you are doing with pp().

    use feature ":5.14"; use warnings FATAL => qw(all); use strict; use Data::Dump qw(dump pp); use Text::CSV_XS; # So original data look something like: my @d = split /\n/, <<'END'; 1,6064.86,85391.25,593.75,13.25 2,6072.17,85392.95,593.79,13.29 3,6078.94,85393.05,593.76,13.26 4,6085.51,85392.22,593.77,13.27 END # I need to insert two lines at the top, two lines at the # bottom, switch columns 2 and 3 around and replace column 5 with colu +mn # 1.I'm using Text::CSV for parsing and also Lava GUI package. So part + of # the code that does parsing looks like this:D my ($t, @t) = (Text::CSV_XS->new); $t->parse($_) && push(@t, [($t->fields())[4,2,1,3,0]]) or die "Could n +ot parse: $_" for @d; # Parse and rearrange unshift @t, ["Top line 1"], ["Top line 2"]; # Top lines push @t, ["Bot line 1"], ["Bot line 2"]; # Bottom lines $t->combine(@$_) && ($_ = $t->string) or die "Cannot combine: ".dump( +$_) for @t; # Combine pp(\@t); # Print result

    Produces

    [ "\"Top line 1\"", "\"Top line 2\"", "13.25,85391.25,6064.86,593.75,1", "13.29,85392.95,6072.17,593.79,2", "13.26,85393.05,6078.94,593.76,3", "13.27,85392.22,6085.51,593.77,4", "\"Bot line 1\"", "\"Bot line 2\"", ]
Re: Parsing Parsing CSV only returns the second line of the file
by Tux (Monsignor) on Sep 01, 2012 at 15:40 UTC

    The outer loop should be written like, more modern and safe (think embedded newlines).

    my $csv_in = Text::CSV_XS->new ({ binary => 1, auto_diag => 1 }); my $csv_out = Text::CSV_XS->new ({ binary => 1, auto_diag => 1, eol => + "\n" }); open my $fh_old, "+<", $file or die Lava::Message ("Can't open origina +l file: $file: $!"); open my $fh_new, ">", $temp or die Lava::Message ("Can't open tempfil +e file: $temp: $!"); $csv->getline ($fh_old); # skip first line. Header?. If empty line, 's +calar <$fh_old>;' is a good option while (my $row = $csv->getline ($fh_old)) { : : $csv_out->print ($fh_new, [ map { $row->[$_] } 0, 2, 1, 3, 0 ]); } $csv_out->print ($fh_new, [ 0, 0.000, 0.000, 0.000 ]); $csv_out->print ($fh_new, [ 0, 0.000, 0.000, 0.000, "END" ]); close $_ for $fh_old, $fh_new;

    Enjoy, Have FUN! H.Merijn

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://991118]
Approved by Athanasius
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (7)
As of 2014-07-13 15:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    When choosing user names for websites, I prefer to use:








    Results (250 votes), past polls