Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

This looks like whitespace in my CSV but doesn't seem to be

by ghenry (Vicar)
on Sep 30, 2012 at 07:59 UTC ( #996472=perlquestion: print w/ replies, xml ) Need Help??
ghenry has asked for the wisdom of the Perl Monks concerning the following question:

Dear all,

I have the following in CSV:

"My Company","Gavin","Henry ","ghenry@ghenry.co.uk"," 1.00"

I have:

my $csv = Text::CSV_XS->new( { binary => 1, eol => $/ } ); open my $io, '<', $file or die "$file: $!"; while ( my $row = $csv->getline($io) ) { my @fields = @$row; next unless $fields[4]; $fields[4] =~ s/\s+//g; print $fields[4], "\n"; }

This looks like whitespace but is never removed. What am I missing? This is soo basic it's driving me mental!

Thanks.

Walking the road to enlightenment... I found a penguin and a camel on the way.....

Comment on This looks like whitespace in my CSV but doesn't seem to be
Select or Download Code
Re: This looks like whitespace in my CSV but doesn't seem to be
by Anonymous Monk on Sep 30, 2012 at 08:09 UTC
    You've got blurry vision
    #!/usr/bin/perl -- use strict; use warnings; use Text::CSV; use Data::Dump; open my $io, '<', \q{ "My Company","Gavin","Henry ","ghenry@ghenry.co.uk"," 1.00" }; my $csv = Text::CSV->new( { binary => 1, eol => $/ } ); while ( my $row = $csv->getline($io) ) { my @fields = @$row; next unless $fields[4]; dd \@fields; $fields[4] =~ s/\s+//g; print $fields[4], "\n"; dd \@fields; } __END__ ["My Company", "Gavin", "Henry ", "ghenry\@ghenry.co.uk", " 1.00"] 1.00 ["My Company", "Gavin", "Henry ", "ghenry\@ghenry.co.uk", "1.00"]

      That gives me:

      [ "My Company", "Gavin", "Henry\xC2\xA0", "ghenry\@ghenry.co.uk", "\xC2\xA1.00", ]

      which isn't whitespace after all. Thanks for the Data::Dump tip. I've always used Data::Dumper. Swapping for that produces:

      $VAR1 = [ 'My Company', 'Gavin', 'Henry ', 'ghenry@ghenry.co.uk', ' 1.00' ]; $VAR1 = [ 'My Company', 'Gavin', 'Henry ', 'ghenry@ghenry.co.uk', ' 1.00' ];

      i.e.

      next unless $fields[4]; print Dumper \@fields; $fields[4] =~ s/\s+//g; print Dumper \@fields;
      Walking the road to enlightenment... I found a penguin and a camel on the way.....
      Fancy a yourname@perl.me.uk? Just ask!!!

        Or is that hex for whitespace? Will check...

        Walking the road to enlightenment... I found a penguin and a camel on the way.....
        Fancy a yourname@perl.me.uk? Just ask!!!

        \xC2\xA0 is a Unicode non-breaking space encoded into UTF-8. If you decode your UTF-8 string into a native Perl Unicode string (see Encode), and then add the /u modifier to your regular expression, that ought to enable \s to remove non-ASCII whitespace.

        perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'
        I don't work with multi-byte characters or UTF-8 files. But Perl has an amazing ability to deal with these things.

        From the open() docs, it could be that all you need to do is open the file in the right encoding mode and that could enable even Perl 5.10.X to work. I don't know for sure, but this idea is worth a try.

        open(my $fh, "<:encoding(UTF-8)", "filename") || die "can't open UTF-8 encoded filename: $!";
Re: This looks like whitespace in my CSV but doesn't seem to be
by tobyink (Abbot) on Sep 30, 2012 at 08:18 UTC

    "This looks like whitespace but is never removed. What am I missing?"

    Text::CSV strips whitespace from unquoted fields by default, but your fields are quoted. Check out String::Strip.

    (Meh, ignore me. I think I got the wrong end of the stick.)

    perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'
Re: This looks like whitespace in my CSV but doesn't seem to be
by Marshall (Prior) on Sep 30, 2012 at 09:29 UTC
    This one way to delete leading and trailing spaces from CSV input.
    #!/usr/bin/perl -w use strict; use Text::CSV; use Data::Dumper; # a core module use Data::Dump qw(pp); open my $io, '<', \q{"My Company","Gavin","Henry ","ghenry@ghenry.co.uk"," 1.00"}; my $csv = Text::CSV->new(); # nothing "fancy" needed here # defaults are fine while ( my $row = $csv->getline($io) ) { my @fields =map { s/^\s*//; #delete leading spaces s/\s*$//; #delete trailing spaces $_; #this is the "return value" from map #map returns the value of the last #statement - without this $_; you #get the zero or non-zero scalar value #of the last s/// statement } @$row; print "Using pp from Data::Dump...\n"; print pp \@fields; print "\n"; print "Using Dumper from Data::Dumper...\n"; print Dumper \@fields; print "\n"; print "5th field is $fields[4]\n"; } __END__ Prints: Using pp from Data::Dump... ["My Company", "Gavin", "Henry", "ghenry\@ghenry.co.uk", "1.00"] Using Dumper from Data::Dumper... $VAR1 = [ 'My Company', 'Gavin', 'Henry', 'ghenry@ghenry.co.uk', '1.00' ]; 5th field is 1.00

      Text:CSV_XS has a native builtin way to delete trailing and leading whitespace:

      my $csv= text::CSV_XS->new ({ binary => 1, allow_whitespace => 1, auto +_diag => 1 });

      But that would not help in this case for two reasons

      • It does not strip whitespace inside quotation, but only surrounding sep_char:
        allow_whitespace When this option is set to true, whitespace (TAB's and SPAC +E's) surrounding the separation character is removed when parsin +g. If either TAB or SPACE is one of the three major characters "sep_char", "quote_char", or "escape_char" it will not be considered whitespace.
      • Thee whitespace stripped is only space or TAB's, and not non-breaking Unicode space stuff

      Spreadsheet::Read however offers to strip leading and trailing whitespace from every field. I could extend that on request to allow it to strip Unicode whitespace too.

      strip If set, "ReadData ()" will remove trailing- and/or leading- whitespace from every field. strip leading strailing ----- ------- --------- 0 n/a n/a 1 strip n/a 2 n/a strip 3 strip strip

      Enjoy, Have FUN! H.Merijn
        This sounds just fine.

        I haven't worked with any CSV files with leading spaces. That idea appears to be uncommon. But your suggestion sounds good.

Re: This looks like whitespace in my CSV but doesn't seem to be
by rpnoble419 (Pilgrim) on Sep 30, 2012 at 21:15 UTC
    Just a point to help you in the future. Does your editor allow you to view your files in Hex? You could easily see that "Whitespace" is not a space and is really a character. This will save you hours of hair pulling.

      I use Vim and with the PBP ~/vimrc additions, with the ptidy shortcut. Will need to check the right setting for Vim.

      Thanks.

      Walking the road to enlightenment... I found a penguin and a camel on the way.....
      Fancy a yourname@perl.me.uk? Just ask!!!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://996472]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (15)
As of 2014-08-27 19:09 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (250 votes), past polls