Re^3: Comparison of the parsing features of CSV (and xSV) modules

[Wally Hartshorn]

Here's an example:

"Smith","John",12/31/1962,"Author of "How to Break Programs" and other books","Bugger"

I'm using a series of (somewhat fragile) regexes to change that to:

"Smith","John",12/31/1962,"Author of ""How to Break Programs"" and other books","Bugger"

[tilly]

There are, of course, going to be boundary cases that don't work as expected as soon as you start playing with allowing undoubled double-quotes inside of a format that expects them doubled. However Text::xSV allows you to define arbitrary filters that it preprocesses text with, and should do a reasonable job on the above with the following filter:

  sub {
    my $line = shift;
    $line =~ s/\r$//;
    $line =~ s/"(.)/""$1/g;
    $line =~ s/"?,"?/,/g;
    return $line;
  }

Yes, there is some fragility, but it should be at least moderately hard to trigger.

[dragonchild]

And, what should the parser do with the following:

"Smith","John",12/31/1962,"Author of "How to Break Programs" and other
+ books,"Bugger"
"Smith","John",12/31/1962,Author of "How to Break Programs" and other 
+books,"Bugger"
"Smith","John",12/31/1962,'Author of "How to Break Programs" and other
+ books,"Bugger"
"Smith","John",12/31/1962,'Author of "How to Break Programs" and other
+ books',"Bugger"

[Wally Hartshorn]

And, what should the parser do with the following:

"Smith","John",12/31/1962,"Author of "How to Break Programs" and other
+ books,"Bugger"
"Smith","John",12/31/1962,"Author of ""How to Break Programs"" and oth
+er books,"Bugger"

"Smith","John",12/31/1962,Author of "How to Break Programs" and other 
+books,"Bugger"
"Smith","John",12/31/1962,Author of ""How to Break Programs"" and othe
+r books,"Bugger"

"Smith","John",12/31/1962,'Author of "How to Break Programs" and other
+ books,"Bugger"
(Reject?)

"Smith","John",12/31/1962,'Author of "How to Break Programs" and other
+ books',"Bugger"
(Reject?)

(I haven't encountered any improperly quoted data, just data that doesn't escape embedded delimiters.)

[dragonchild]

What about the following:

abcd,"efgh,"ijkl,"mnop",qrst

Is that malformed or is that meant to be

abcd,"efgh,""ijkl,""mnop",qrst

The issue is that there are too many edge cases for a general-purpose parser to handle. I'm coming up with a bunch and I'm not even trying hard.