Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Setting end position for the regexp engine

by bojinlund (Monsignor)
on Sep 18, 2013 at 19:17 UTC ( [id://1054726]=perlmeditation: print w/replies, xml ) Need Help??

Hej!

Background

I often need to do small and incomplete parsers. An example of such parser is included at the and.

The principles of this parser

Read the file into $text. Divide $text in parts. The parts are represented in @part. For each part there is [ $type, $first, $last ]. $first and $last are positions in $text.

The parsing is done by the sub parse. It is controlled by @spec, which contains a number of [ $reg_exp, $type ]. All the $reg_exp start with \G. The $reg_exp are tried in order. After a match the tries starts with the first $reg_exp. The last $reg_exp must match so much that one of the other $reg_exp can match and the $type = '??'.

In this and in many cases the parsing is done step-by-step. In the example the first step is locating strings ( 'sq' and 'dq' ) and comments ( 'cn' and 'cb' ). The second step is then, in the parts with $type eq '??', locating names ( 'na' ), digits ( 'di' ) and reserved words ( 'rw' ).

Discussion

By using the function pos() in combination with the \G zero-width assertion you can control where the regexp engine starts. The sub parser can been simplified if it was possible to set the position where the regexp engine stops.

Are there any easy way, with the current Perl, to control where the engine should stop?

Or is it feasible to add a new feature to Perl. Possibilities could be:

  • A new function stop_pos().
  • Extend the binary operator =~, which binds a scalar expression to a pattern match, with parameters like =~[$start, $stop].
  • Add a string slice $string.slice( $start, $stop), which can be used together with =~.
  • Let function substr return a lvalue which could be store in a (new type of?) scalar. The lvalue can then be used together with =~.

Example of parser

#!/usr/bin/perl -w use strict; use warnings; sub parse { my ( $txt_ref, $part_aref, $spec_aref ) = @_; my ( $type, $first, $last ); # for the current part # used in the current matching my $cur_txt_ref; my $cur_sub_str; my $cur_pos0; # start of $cur_sub_str in $$txt_r +ef my @part; # result from the parsing my $do_part = sub { my $end_last_match = 0; MATCH: { for my $spec (@$spec_aref) { my ( $reg_exp, $type ) = @$spec; if ( $$cur_txt_ref =~ m{$reg_exp}gcm ) { push @part, [ $type, $-[0] + $cur_pos0, $+[0] + $c +ur_pos0 ]; $end_last_match = $+[0]; redo MATCH; } } last MATCH; } if ( $end_last_match < length $$cur_txt_ref ) { warn 'ERROR: Stopped before string end at pos: ', "$cur_pos0 + $end_last_match\n<", substr( $$cur_txt_ref, $end_last_match ), '>'; } }; if ( not defined $part_aref ) { # use the whole $$txt_ref $cur_txt_ref = $txt_ref; $cur_pos0 = 0; $do_part->(); return \@part; } $cur_txt_ref = \$cur_sub_str; # use a substring from $$txt_re +f map { ( $type, $first, $last ) = @$_; if ( $type eq '??' ) { $cur_sub_str = substr( $$txt_ref, $first, $last - $first ) +; $cur_pos0 = $first; $do_part->(); } else { push @part, $_; } } @$part_aref; return \@part; } my @spec_1 = ( [ qr{\G'[^'\\]*(?:\\.[^'\\]*)*'}, 'sq' ], [ qr{\G"[^"\\]*(?:\\.[^"\\]*)*"}, 'dq' ], [ qr{\G//[^\n]*[\n]?}, 'cn' ], [ qr{\G/[*](?:[^*]*|[*]+[^/*]*)*[*]/}, 'cb' ], [ qr{\G(?:[^'"/]+|[/][^'"/*])+}, '??' ], ); my $dig1 = qr{-?\d+\.\d*}; my $dig2 = qr{-?\d*\.\d+}; my $dig3 = qr{-?\d+}; my $dig4 = qr{E-?\d+}; my $digit = qr{(?:$dig1|$dig2|$dig3)$dig4?}; my @spec_2 = ( [ qr{\G(?:var|alert)}, 'rw' ], [ qr{\G$digit}, 'di' ], [ qr{\G[_a-zA-Z0-9.\$]+}, 'na' ], [ qr{\G(?:[^_a-zA-Z0-9.\$\d]+|[\n\s]+)+}, '??' ], ); sub to_string_part_aref { my ( $txt_ref, $part_aref ) = @_; return join '', map { my ( $type, $first, $last ) = @$_; if ( $type eq '??' ) { substr( $$txt_ref, $first, $last - $first ); } else { "<$type>" . substr( $$txt_ref, $first, $last - $first ) . "</$type>"; } } @{$part_aref}; } my $text = do { local $/; <DATA> }; my $text_ref = \$text; my $part_ref_1 = parse( $text_ref, undef, \@spec_1 ); my $part_ref_2 = parse( $text_ref, $part_ref_1, \@spec_2 ); warn to_string_part_aref( $text_ref, $part_ref_2 ); exit 0; __DATA__ // This is a single-line comment var x = 4; // Single /* Multiple-line comment that can span any number of lines */ /* This is a multi-line comment // Still a multi-line comment */ /* Stop code var x = 4; var y = 5; /* Bug? * x = "cool"; End Stop code */ // This is a single-line comment /* ...still a single-line comment 'string\' // still a string'; // comment /* not-a-nested-comment var = 0.5; // comment */* still-a-comment ' /**/ string ' /* "comment..." // still-a-comment */ alert('This isn\'t a comment!'); /\/* this isn't a comment! */; //* comment /* //a comment... // still-a-comment 12345 "Foo /bar/ "" */ /*//Boo*/ /*/**/

Replies are listed 'Best First'.
Re: Setting end position for the regexp engine
by BrowserUk (Patriarch) on Sep 18, 2013 at 21:19 UTC
    Let function substr return a lvalue which could be store in a (new type of?) scalar. The lvalue can then be used together with =~.

    That is already possible.

    C:\test>p1 $s = 'This is a string (with a bit in the middle) before the string en +ds';; $r = \substr $s, 18, 24;; print $$r;; with a bit in the middle $$r =~ s[([a-z]+)][\U$1]g;; print $s;; This is a string (WITH A BIT IN THE MIDDLE) before the string ends

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      BrowserUK thank you very much for the answer!

      I have made two new versions of the example parser using the LVALUE returned by substr.

      In version 2 I use [ $type, $first, $length ] to represent a part in @part.

      In version 3 I use [ $type, $sub_txt_ref ], where $sub_txt_ref is a LVALUE returned by substr.

      The scalar type LVALUE is new form me. Where can I read about it?

      Example of parser version 2

      Example of parser version 3

      Is updated! Removed the parameter $txt_ref.

        ref returns LVALUE for a reference to a PVLV, a scalar subtype. You can read about it a bit here. It's basically a scalar that's going to have magic associated with it, and it has a few extra fields for storage the magic can use.


        Magic, among other things, allows one to attach a getter and fetcher to a variable.

        You might think that

        substr($_, 3, 2) = 'foo';

        gets converted to

        substr($_, 3, 2, 'foo');

        by the compiler, but it isn't. substr is truly called as is. This allows the following to work

        my $ref = \substr($_, 3, 2); $$ref = 'foo';

        That means substr must return a magical scalar.


        There are more than one type of scalar. Some can hold an integer, some can hold a string, some can hold both. Perl automatically upgrades a scalar when necessary.

        Magical scalars require extra fields to store information about what magic is attached to the scalar. The most basic scalar subtype capable of being magical is the PVMG, but there is also PVLV. The PVLV is a PVMG with four extra fields: TYPE, TARGOFF, TARGLEN and TARG.

        A instance of substr in an lvalue context returns a PVLV. It uses TARG, TARGOFF and TARGLEN to store the three arguments passed to substr[1].

        >perl -MDevel::Peek -e"$_ = 'abcdef'; my $ref = \substr($_, 3, 2); Dum +p($$ref);" SV = PVLV(0x4d3d24) at 0x2cb29c <--- PVLV REFCNT = 1 FLAGS = (TEMP,GMG,SMG) <--- "get" magic and "set" magic IV = 0 NV = 0 PV = 0 MAGIC = 0x49f09c MG_VIRTUAL = &PL_vtbl_substr <--- Function pointers for magic MG_TYPE = PERL_MAGIC_substr(x) <--- x = substr magic TYPE = x <--- x = substr magic TARGOFF = 3 <--- For "x", start offset of substring TARGLEN = 2 <--- For "x", length of substring TARG = 0x4a8414 <--- For "x", addr of associated scalar ($_) FLAGS = 0 SV = PV(0x2c8a6c) at 0x4a8414 <--- Dump of associated scalar ($_) REFCNT = 2 FLAGS = (POK,pPOK) PV = 0x2cab94 "abcdef"\0 CUR = 6 LEN = 12

        When you try to fetch from $$ref, the associated get magic first does something like $$ref = substr($$TARG, $TARGOFF, $TARGLEN);.

        After you assign to $$ref, the associated "set" magic effectively does substr($$TARG, $TARGOFF, $TARGLEN, $$ref);.


        Notes

        1. Missing and negative arguments are resolved before being assigned.
        The scalar type LVALUE is new form me. Where can I read about it?

        Um. Beyond perldoc:substr, I'm not sure.

        It has been a part of my lexicon for so long, I've forgotten how or when I first learnt of it.

        And of course there is this, where the hereditary peer hereabouts ... (draw your own conclusions).


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Setting end position for the regexp engine (*ACCEPT)
by Anonymous Monk on Sep 18, 2013 at 22:38 UTC

    I never used it but (*ACCEPT) sounds like it might be like stop_pos

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlmeditation [id://1054726]
Approved by keszler
Front-paged by ww
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others cooling their heels in the Monastery: (2)
As of 2024-03-19 06:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found