Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?

Re^2: Speeds vs functionality

by oiskuu (Hermit)
on Jul 30, 2014 at 08:52 UTC ( #1095583=note: print w/replies, xml ) Need Help??

in reply to Re: Speeds vs functionality
in thread Speeds vs functionality

Looking at cx_Parse + stuff, some thoughts and questions arise:

  • Is it necessary to fiddle with the cache? Just undef _cache on perl side to enforce parameter changes; this ought to be a rare event? Stashing an opaque C struct avoids needless copying.
  • Then what about unicode whitespace characters?
  • quote_char, escape_char, etc., could be ints and default to -1 when undefined. Easier to test against. However ...
  • Have you tried writing this thing as an fsm?
  • enum { TXT, BIN, WSPACE, UTF8, QUOT, ESC, SEP, CR, NL_EOLX, ..., NN }; enum { EXFIELD, INFIELD = 1*NN, INQUOT = 2*NN, CRSEEN = 3*NN } state; while ((c = getc()) != EOF) { int ctype = cached->xlat[c]; if () ... /* perhaps peel the most likely case(s) */ switch (state + ctype) { case WSPACE: continue; /* nop: allow_whitespace test in xlat[] */ case BIN: error(); /* again, resolved when constructing xlat[] */ case TXT: state = INFIELD; putc(c); continue; case INFIELD+TXT: case INQUOT+TXT: case INQUOT+SEP: ... putc(c); ... case UTF8: case INFIELD+UTF8: ...accumulate/xlat... case CRSEEN+NL_EOLX: ...; state = 0; continue; case CRSEEN+...: error(); default: error(); } ...
    Or possibly:
    enum { EXFIELD, INFIELD = 0x100, INQUOT = 0x200, CRS = 0x300 } sta +te; ... int action = cached->xlat[state + c]; decode(action); ...

Ultimately, the (handful of) UTF sequences may also be resolved by walking trie-like state tables.

Replies are listed 'Best First'.
Re^3: Speeds vs functionality
by Tux (Abbot) on Jul 30, 2014 at 12:21 UTC

    The cache, as implemented currently, was implemented to achief a boost of (iirc) about 25%. It is needed to reduce the access to the object (the $self hash), as those lookups are very very expensive.

    Unicode whitespace isn't important for this parser, as it is no special "character", unless it being the separator, the quotation or the escape character. Unicode whitespace will just end up being binary.

    XS is not PP :) Those characters could be int indeed, but that would probably mean that the whole parser (written in 1998 and modified/extended over time) has to be rewritten. It /might/ be worth the effort in the end, but I do not have the time to start that experiment.

    Never tried fsm (unless the current state-machine already is an FSM). I simplified the parser as I got it when I took over maint. Over time a lot of bugs were fixed and new (required and requested) features where added.

    update: added remark about FSM

    Enjoy, Have FUN! H.Merijn
      Is there any reason stopping you for keeping the parser state as a persistent C struct?

      Correct me if I am wrong: currently, the state is keep exclusively on the Perl side and the cache is a (ugly) hack to be able to regenerate the C struct faster.

      Why don't just store the state on the C side and keep it as a pointer inside a IV? That's what most XS libs do and I am sure it would improve the parser speed a lot, and at the same time simplify the code!

      Are the module users allowed to modify the object hash directly?

        I am keeping it as a cache, because the user is allowed to alter the behavior of the parser between parses. And that is useful. I also agree that the cache-dealing code has by now grown out to a horrid state, and I was already playing with the idea of replacing it with a single PV that holds a memcpy of the csv struct (which might need some extending then).

        In earlier days, the end user was allowed to alter the hash. I now only allow (reliable) changes through the method calls. That implies that I in theory can move all code to XS.

        I however am not yet sure if that would simplify the code, though it probably will :)

        Enjoy, Have FUN! H.Merijn

      I believe Modern Perl should have a core module that can easily parse these simple Unicode CSV records. It should handle them in any character encoding scheme of Unicode:  UTF-8, UTF-16, or UTF-32. And it should handle the Unicode byte order mark seamlessly.

      Why not?

      🎥12 Years a Slave🎥🎬2013🎬3🎬9🎬🎥🎥🎥 Steve McQueen🎥
      🎥Argo🎥🎬2012🎬3🎬7🎬🎥🎥🎥 Ben Affleck🎥
      🎥The Artist🎥🎬2012🎬5🎬10🎬🎥🎥🎥 Michel Hazanavicius🎥
      🎥The King's Speech🎥🎬2010🎬4🎬12🎬🎥🎥🎥 Tom Hooper🎥
      🎥The Hurt Locker🎥🎬2009🎬6🎬9🎬🎥🎥🎥 Kathryn Bigelow🎥
      🎥Slumdog Millionaire🎥🎬2008🎬8🎬10🎬🎥🎥🎥 Danny Boyle🎥
      🎥No Country for Old Men🎥🎬2007🎬4🎬8🎬🎥🎥🎥 Joel Coen
      🎥🎥 Ethan Coen🎥
      🎥The Departed🎥🎬2006🎬4🎬5🎬🎥🎥🎥 Martin Scorsese🎥

      sep_char	🎬	U+1F3AC CLAPPER BOARD (UTF-8: F0 9F 8E AC)
      quote_char	🎥	U+1F3A5 MOVIE CAMERA  (UTF-8: F0 9F 8E A5)
      escape_char	🎥	U+1F3A5 MOVIE CAMERA  (UTF-8: F0 9F 8E A5)
      "12 Years a Slave",2013,3,9,"🎥 Steve McQueen"
      "Argo",2012,3,7,"🎥 Ben Affleck"
      "The Artist",2012,5,10,"🎥 Michel Hazanavicius"
      "The King's Speech",2010,4,12,"🎥 Tom Hooper"
      "The Hurt Locker",2009,6,9,"🎥 Kathryn Bigelow"
      "Slumdog Millionaire",2008,8,10,"🎥 Danny Boyle"
      "No Country for Old Men",2007,4,8,"🎥 Joel Coen
      🎥 Ethan Coen"
      "The Departed",2006,4,5,"🎥 Martin Scorsese"

      I recognize that the current XS core module for parsing CSV records, Text::CSV_XS (marvelously maintained by Tux), may not be the right module to use as the basis for a new, fully Unicode-capable module. But because Perl's native Unicode capabilities exceed those of most other programming languages, Perl should have a proper FSM-based Unicode CSV parser, even if it's pure Perl and not XS.

      I long ago accepted that Unicode conformance and comparative slowness go hand in hand 👫. So what? Look what you're trading a few seconds here and there for:  the technological foundation of World Peace ☮ and Universal Love 💕.

      UPDATE:  Removed references to core module. I don't care about that. I just want a Unicode-capable Perl CSV module.

        Text::CSV_PP is able to parse that text, at least in UTF-8.

        use v5.12;
        use warnings;
        use utf8::all;
        use Text::CSV_PP;
        my $csv = Text::CSV_PP->new ( 
            { binary      => 1 , 
              quote_char  => '🎥' ,
              escape_char => '🎥' ,
              sep_char    => '🎬'  } )
          or die "Cannot use CSV_PP: "
           .Text::CSV_PP->error_diag ();
        my @rows;
        my $fh = *DATA;
        while ( my $row = $csv->getline( $fh ) ) {
                 push @rows, $row;
        $csv->eof or $csv->error_diag();
        for ( @rows ) {
            printf("%-25s%s\n", $_->[0], $_->[4]);
        🎥12 Years a Slave🎥🎬2013🎬3🎬9🎬🎥🎥🎥 Steve McQueen🎥
        🎥Argo🎥🎬2012🎬3🎬7🎬🎥🎥🎥 Ben Affleck🎥
        🎥The Artist🎥🎬2012🎬5🎬10🎬🎥🎥🎥 Michel Hazanavicius🎥
        🎥The King's Speech🎥🎬2010🎬4🎬12🎬🎥🎥🎥 Tom Hooper🎥
        🎥The Hurt Locker🎥🎬2009🎬6🎬9🎬🎥🎥🎥 Kathryn Bigelow🎥
        🎥Slumdog Millionaire🎥🎬2008🎬8🎬10🎬🎥🎥🎥 Danny Boyle🎥
        🎥No Country for Old Men🎥🎬2007🎬4🎬8🎬🎥🎥🎥 Joel Coen 🎥🎥 Ethan Coen🎥
        🎥The Departed🎥🎬2006🎬4🎬5🎬🎥🎥🎥 Martin Scorsese🎥


        Film                     Director
        12 Years a Slave         🎥 Steve McQueen
        Argo                     🎥 Ben Affleck
        The Artist               🎥 Michel Hazanavicius
        The King's Speech        🎥 Tom Hooper
        The Hurt Locker          🎥 Kathryn Bigelow
        Slumdog Millionaire      🎥 Danny Boyle
        No Country for Old Men   🎥 Joel Coen 🎥 Ethan Coen
        The Departed             🎥 Martin Scorsese

        There are already at least a few CSV parsing modules on CPAN that don't just wrap Text::CSV_XS. A pure-Perl CSV parser is likely going to "just work" when given a file handle with the right encoding declared and separator/quote/escape strings properly decoded.

        Parse::CSV and Text::xSV are the first two I would try. My expectation is that both will handle utf-8 just fine. And if either doesn't, I suspect that fixing that problem won't be difficult.

        - tye        

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1095583]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (6)
As of 2018-02-24 15:02 GMT
Find Nodes?
    Voting Booth?
    When it is dark outside I am happiest to see ...

    Results (310 votes). Check out past polls.