|
in reply to Re: Speeds vs functionality
in thread Speeds vs functionality
Looking at cx_Parse + stuff, some thoughts and questions arise:
- Is it necessary to fiddle with the cache? Just undef _cache on perl side to enforce parameter changes; this ought to be a rare event? Stashing an opaque C struct avoids needless copying.
- Then what about unicode whitespace characters?
- quote_char, escape_char, etc., could be ints and default to -1 when undefined. Easier to test against. However ...
- Have you tried writing this thing as an fsm?
enum { TXT, BIN, WSPACE, UTF8, QUOT, ESC, SEP, CR, NL_EOLX, ..., NN };
enum { EXFIELD, INFIELD = 1*NN, INQUOT = 2*NN, CRSEEN = 3*NN } state;
while ((c = getc()) != EOF) {
int ctype = cached->xlat[c];
if () ... /* perhaps peel the most likely case(s) */
switch (state + ctype) {
case WSPACE: continue; /* nop: allow_whitespace test in xlat[] */
case BIN: error(); /* again, resolved when constructing xlat[] */
case TXT: state = INFIELD; putc(c); continue;
case INFIELD+TXT: case INQUOT+TXT: case INQUOT+SEP: ... putc(c);
...
case UTF8: case INFIELD+UTF8: ...accumulate/xlat...
case CRSEEN+NL_EOLX: ...; state = 0; continue;
case CRSEEN+...: error();
default: error();
}
...
Or possibly:
enum { EXFIELD, INFIELD = 0x100, INQUOT = 0x200, CRS = 0x300 } sta
+te;
...
int action = cached->xlat[state + c];
decode(action);
...
Ultimately, the (handful of) UTF sequences may also be resolved by walking trie-like state tables.
|
Re^3: Speeds vs functionality
by Tux (Canon) on Jul 30, 2014 at 12:21 UTC
|
The cache, as implemented currently, was implemented to achief a boost of (iirc) about 25%. It is needed to reduce the access to the object (the $self hash), as those lookups are very very expensive.
Unicode whitespace isn't important for this parser, as it is no special "character", unless it being the separator, the quotation or the escape character. Unicode whitespace will just end up being binary.
XS is not PP :) Those characters could be int indeed, but that would probably mean that the whole parser (written in 1998 and modified/extended over time) has to be rewritten. It /might/ be worth the effort in the end, but I do not have the time to start that experiment.
Never tried fsm (unless the current state-machine already is an FSM). I simplified the parser as I got it when I took over maint. Over time a lot of bugs were fixed and new (required and requested) features where added.
update: added remark about FSM
Enjoy, Have FUN! H.Merijn
| [reply]
[d/l] |
|
|
Is there any reason stopping you for keeping the parser state as a persistent C struct?
Correct me if I am wrong: currently, the state is keep exclusively on the Perl side and the cache is a (ugly) hack to be able to regenerate the C struct faster.
Why don't just store the state on the C side and keep it as a pointer inside a IV? That's what most XS libs do and I am sure it would improve the parser speed a lot, and at the same time simplify the code!
Are the module users allowed to modify the object hash directly?
| [reply] |
|
|
I am keeping it as a cache, because the user is allowed to alter the behavior of the parser between parses. And that is useful. I also agree that the cache-dealing code has by now grown out to a horrid state, and I was already playing with the idea of replacing it with a single PV that holds a memcpy of the csv struct (which might need some extending then).
In earlier days, the end user was allowed to alter the hash. I now only allow (reliable) changes through the method calls. That implies that I in theory can move all code to XS.
I however am not yet sure if that would simplify the code, though it probably will :)
Enjoy, Have FUN! H.Merijn
| [reply] |
|
|
I believe Modern Perl should have a core module that can easily parse these simple Unicode CSV records. It should handle them in any character encoding scheme of Unicode: UTF-8, UTF-16, or UTF-32. And it should handle the Unicode byte order mark seamlessly.
Why not?
🎥Film🎥🎬🎥Year🎥🎬🎥Awards🎥🎬🎥Nominations🎥🎬🎥Director🎥
🎥12 Years a Slave🎥🎬2013🎬3🎬9🎬🎥🎥🎥 Steve McQueen🎥
🎥Argo🎥🎬2012🎬3🎬7🎬🎥🎥🎥 Ben Affleck🎥
🎥The Artist🎥🎬2012🎬5🎬10🎬🎥🎥🎥 Michel Hazanavicius🎥
🎥The King's Speech🎥🎬2010🎬4🎬12🎬🎥🎥🎥 Tom Hooper🎥
🎥The Hurt Locker🎥🎬2009🎬6🎬9🎬🎥🎥🎥 Kathryn Bigelow🎥
🎥Slumdog Millionaire🎥🎬2008🎬8🎬10🎬🎥🎥🎥 Danny Boyle🎥
🎥No Country for Old Men🎥🎬2007🎬4🎬8🎬🎥🎥🎥 Joel Coen
🎥🎥 Ethan Coen🎥
🎥The Departed🎥🎬2006🎬4🎬5🎬🎥🎥🎥 Martin Scorsese🎥
sep_char 🎬 U+1F3AC CLAPPER BOARD (UTF-8: F0 9F 8E AC)
quote_char 🎥 U+1F3A5 MOVIE CAMERA (UTF-8: F0 9F 8E A5)
escape_char 🎥 U+1F3A5 MOVIE CAMERA (UTF-8: F0 9F 8E A5)
"Film","Year","Awards","Nominations","Director"
"12 Years a Slave",2013,3,9,"🎥 Steve McQueen"
"Argo",2012,3,7,"🎥 Ben Affleck"
"The Artist",2012,5,10,"🎥 Michel Hazanavicius"
"The King's Speech",2010,4,12,"🎥 Tom Hooper"
"The Hurt Locker",2009,6,9,"🎥 Kathryn Bigelow"
"Slumdog Millionaire",2008,8,10,"🎥 Danny Boyle"
"No Country for Old Men",2007,4,8,"🎥 Joel Coen
🎥 Ethan Coen"
"The Departed",2006,4,5,"🎥 Martin Scorsese"
I recognize that the current XS core module for parsing CSV records, Text::CSV_XS (marvelously maintained by Tux), may not be the right module to use as the basis for a new, fully Unicode-capable module. But because Perl's native Unicode capabilities exceed those of most other programming languages, Perl should have a proper FSM-based Unicode CSV parser, even if it's pure Perl and not XS.
I long ago accepted that Unicode conformance and comparative slowness go hand in hand 👫. So what? Look what you're trading a few seconds here and there for: the technological foundation of World Peace ☮ and Universal Love 💕.
UPDATE: Removed references to core module. I don't care about that. I just want a Unicode-capable Perl CSV module.
| [reply] |
|
|
use v5.12;
use warnings;
use utf8::all;
use Text::CSV_PP;
my $csv = Text::CSV_PP->new (
{ binary => 1 ,
quote_char => '🎥' ,
escape_char => '🎥' ,
sep_char => '🎬' } )
or die "Cannot use CSV_PP: "
.Text::CSV_PP->error_diag ();
my @rows;
my $fh = *DATA;
while ( my $row = $csv->getline( $fh ) ) {
push @rows, $row;
}
$csv->eof or $csv->error_diag();
for ( @rows ) {
printf("%-25s%s\n", $_->[0], $_->[4]);
}
__DATA__
🎥Film🎥🎬🎥Year🎥🎬🎥Awards🎥🎬🎥Nominations🎥🎬🎥Director🎥
🎥12 Years a Slave🎥🎬2013🎬3🎬9🎬🎥🎥🎥 Steve McQueen🎥
🎥Argo🎥🎬2012🎬3🎬7🎬🎥🎥🎥 Ben Affleck🎥
🎥The Artist🎥🎬2012🎬5🎬10🎬🎥🎥🎥 Michel Hazanavicius🎥
🎥The King's Speech🎥🎬2010🎬4🎬12🎬🎥🎥🎥 Tom Hooper🎥
🎥The Hurt Locker🎥🎬2009🎬6🎬9🎬🎥🎥🎥 Kathryn Bigelow🎥
🎥Slumdog Millionaire🎥🎬2008🎬8🎬10🎬🎥🎥🎥 Danny Boyle🎥
🎥No Country for Old Men🎥🎬2007🎬4🎬8🎬🎥🎥🎥 Joel Coen 🎥🎥 Ethan Coen🎥
🎥The Departed🎥🎬2006🎬4🎬5🎬🎥🎥🎥 Martin Scorsese🎥
Output:
Film Director
12 Years a Slave 🎥 Steve McQueen
Argo 🎥 Ben Affleck
The Artist 🎥 Michel Hazanavicius
The King's Speech 🎥 Tom Hooper
The Hurt Locker 🎥 Kathryn Bigelow
Slumdog Millionaire 🎥 Danny Boyle
No Country for Old Men 🎥 Joel Coen 🎥 Ethan Coen
The Departed 🎥 Martin Scorsese
| [reply] |
|
|
|
|
|
|
|
|
There are already at least a few CSV parsing modules on CPAN that don't just wrap Text::CSV_XS. A pure-Perl CSV parser is likely going to "just work" when given a file handle with the right encoding declared and separator/quote/escape strings properly decoded.
Parse::CSV and Text::xSV are the first two I would try. My expectation is that both will handle utf-8 just fine. And if either doesn't, I suspect that fixing that problem won't be difficult.
| [reply] |
|
|
|
|
| |
|
|
|
| [reply] |
|
|
