Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^2: Setting end position for the regexp engine using LVALUE

by bojinlund (Curate)
on Sep 19, 2013 at 07:37 UTC ( #1054793=note: print w/ replies, xml ) Need Help??


in reply to Re: Setting end position for the regexp engine
in thread Setting end position for the regexp engine

BrowserUK thank you very much for the answer!

I have made two new versions of the example parser using the LVALUE returned by substr.

In version 2 I use [ $type, $first, $length ] to represent a part in @part.

In version 3 I use [ $type, $sub_txt_ref ], where $sub_txt_ref is a LVALUE returned by substr.

The scalar type LVALUE is new form me. Where can I read about it?

Example of parser version 2

use strict; use warnings; sub parse { my ( $txt_ref, $part_aref, $spec_aref ) = @_; if ( not defined $part_aref ) { # use the whole $$txt_ref $part_aref = [ [ '??', 0, length($$txt_ref) ] ]; } my @part; # result from the parsing for my $part (@$part_aref) { my ( $type, $first, $length ) = @$part; if ( $type ne '??' ) { push @part, $part; next; } my $sub_txt_ref = \substr( $$txt_ref, $first, $length ); my $end_last_match = 0; MATCH: { for my $spec (@$spec_aref) { my ( $reg_exp, $type ) = @$spec; if ( $$sub_txt_ref =~ m{$reg_exp}gcm ) { push @part, [ $type, $-[0] + $first, $+[0] - $-[0] + ]; $end_last_match = $+[0]; redo MATCH; } } } if ( $end_last_match < $length ) { warn 'ERROR: Stopped before string end at pos: ', "$first + $end_last_match\n<", substr( $$sub_txt_ref, $end_last_match ), '>'; } } return \@part; } my @spec_1 = ( [ qr{\G'[^'\\]*(?:\\.[^'\\]*)*'}, 'sq' ], [ qr{\G"[^"\\]*(?:\\.[^"\\]*)*"}, 'dq' ], [ qr{\G//[^\n]*[\n]?}, 'cn' ], [ qr{\G/[*](?:[^*]*|[*]+[^/*]*)*[*]/}, 'cb' ], [ qr{\G(?:[^'"/]+|[/][^'"/*])+}, '??' ], ); my $dig1 = qr{-?\d+\.\d*}; my $dig2 = qr{-?\d*\.\d+}; my $dig3 = qr{-?\d+}; my $dig4 = qr{E-?\d+}; my $digit = qr{(?:$dig1|$dig2|$dig3)$dig4?}; my @spec_2 = ( [ qr{\G(?:var|alert)}, 'rw' ], [ qr{\G$digit}, 'di' ], [ qr{\G[_a-zA-Z0-9.\$]+}, 'na' ], [ qr{\G(?:[^_a-zA-Z0-9.\$\d]+|[\n\s]+)+}, '??' ], ); sub to_string_part_aref { my ( $txt_ref, $part_aref ) = @_; return join '', map { my ( $type, $first, $length ) = @$_; if ( $type eq '??' ) { substr( $$txt_ref, $first, $length ); } else { "<$type>" . substr( $$txt_ref, $first, $length ) . "</$typ +e>"; } } @{$part_aref}; } my $text = do { local $/; <DATA> }; my $text_ref = \$text; my $part_ref_1 = parse( $text_ref, undef, \@spec_1 ); my $part_ref_2 = parse( $text_ref, $part_ref_1, \@spec_2 ); warn to_string_part_aref( $text_ref, $part_ref_2 ); __DATA__ // This is a single-line comment var x = 4; // Single /* Multiple-line comment that can span any number of lines */ /* This is a multi-line comment // Still a multi-line comment */ /* Stop code var x = 4; var y = 5; /* Bug? * x = "cool"; End Stop code */ // This is a single-line comment /* ...still a single-line comment 'string\' // still a string'; // comment /* not-a-nested-comment var = 0.5; // comment */* still-a-comment ' /**/ string ' /* "comment..." // still-a-comment */ alert('This isn\'t a comment!'); /\/* this isn't a comment! */; //* comment /* //a comment... // still-a-comment 12345 "Foo /bar/ "" */ /*//Boo*/ /*/**/

Example of parser version 3

Is updated! Removed the parameter $txt_ref.

use strict; use warnings; sub first_part_aref { my ($txt_ref) = @_; my $sub_txt_ref = \substr( $$txt_ref, 0 ); return [ [ '??', $sub_txt_ref ] ]; } sub parse { my ( $part_aref, $spec_aref ) = @_; my @part; # result from the parsing for my $part (@$part_aref) { my ( $type, $sub_txt_ref ) = @$part; if ( $type ne '??' ) { push @part, $part; next; } my $end_last_match = 0; MATCH: { for my $spec (@$spec_aref) { my ( $reg_exp, $type ) = @$spec; if ( $$sub_txt_ref =~ m{$reg_exp}gcm ) { my $sub_sub_txt_ref = \substr( $$sub_txt_ref, $-[0], $+[0] - $-[0] ); push @part, [ $type, $sub_sub_txt_ref ]; $end_last_match = $+[0]; redo MATCH; } } } if ( $end_last_match < length $$sub_txt_ref ) { warn 'ERROR: Stopped before string end at pos: ', "$end_last_match\n<", substr( $$sub_txt_ref, $end_last_match ), '>'; } } return \@part; } my @spec_1 = ( [ qr{\G'[^'\\]*(?:\\.[^'\\]*)*'}, 'sq' ], [ qr{\G"[^"\\]*(?:\\.[^"\\]*)*"}, 'dq' ], [ qr{\G//[^\n]*[\n]?}, 'cn' ], [ qr{\G/[*](?:[^*]*|[*]+[^/*]*)*[*]/}, 'cb' ], [ qr{\G(?:[^'"/]+|[/][^'"/*])+}, '??' ], ); my $dig1 = qr{-?\d+\.\d*}; my $dig2 = qr{-?\d*\.\d+}; my $dig3 = qr{-?\d+}; my $dig4 = qr{E-?\d+}; my $digit = qr{(?:$dig1|$dig2|$dig3)$dig4?}; my @spec_2 = ( [ qr{\G(?:var|alert)}, 'rw' ], [ qr{\G$digit}, 'di' ], [ qr{\G[_a-zA-Z0-9.\$]+}, 'na' ], [ qr{\G(?:[^_a-zA-Z0-9.\$\d]+|[\n\s]+)+}, '??' ], ); sub to_string_part_aref { my ($part_aref) = @_; return join '', map { my ( $type, $sub_txt_ref ) = @$_; $type ne '??' ? "<$type>" : (), $$sub_txt_ref, $type ne '??' ? "</$type>" : (); } @{$part_aref}; } my $text = do { local $/; <DATA> }; my $part_ref_0 = first_part_aref( \$text ); my $part_ref_1 = parse( $part_ref_0, \@spec_1 ); my $part_ref_2 = parse( $part_ref_1, \@spec_2 ); warn to_string_part_aref($part_ref_2); __DATA__ // This is a single-line comment var x = 4; // Single /* Multiple-line comment that can span any number of lines */ /* This is a multi-line comment // Still a multi-line comment */ /* Stop code var x = 4; var y = 5; /* Bug? * x = "cool"; End Stop code */ // This is a single-line comment /* ...still a single-line comment 'string\' // still a string'; // comment /* not-a-nested-comment var = 0.5; // comment */* still-a-comment ' /**/ string ' /* "comment..." // still-a-comment */ alert('This isn\'t a comment!'); /\/* this isn't a comment! */; //* comment /* //a comment... // still-a-comment 12345 "Foo /bar/ "" */ /*//Boo*/ /*/**/


Comment on Re^2: Setting end position for the regexp engine using LVALUE
Select or Download Code
Re^3: Setting end position for the regexp engine using LVALUE
by BrowserUk (Pope) on Sep 19, 2013 at 07:54 UTC
    The scalar type LVALUE is new form me. Where can I read about it?

    Um. Beyond perldoc:substr, I'm not sure.

    It has been a part of my lexicon for so long, I've forgotten how or when I first learnt of it.

    And of course there is this, where the hereditary peer hereabouts ... (draw your own conclusions).


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re^3: Setting end position for the regexp engine using LVALUE
by ikegami (Pope) on Sep 19, 2013 at 23:31 UTC

    ref returns LVALUE for a reference to a PVLV, a scalar subtype. You can read about it a bit here. It's basically a scalar that's going to have magic associated with it, and it has a few extra fields for storage the magic can use.


    Magic, among other things, allows one to attach a getter and fetcher to a variable.

    You might think that

    substr($_, 3, 2) = 'foo';

    gets converted to

    substr($_, 3, 2, 'foo');

    by the compiler, but it isn't. substr is truly called as is. This allows the following to work

    my $ref = \substr($_, 3, 2); $$ref = 'foo';

    That means substr must return a magical scalar.


    There are more than one type of scalar. Some can hold an integer, some can hold a string, some can hold both. Perl automatically upgrades a scalar when necessary.

    Magical scalars require extra fields to store information about what magic is attached to the scalar. The most basic scalar subtype capable of being magical is the PVMG, but there is also PVLV. The PVLV is a PVMG with four extra fields: TYPE, TARGOFF, TARGLEN and TARG.

    A instance of substr in an lvalue context returns a PVLV. It uses TARG, TARGOFF and TARGLEN to store the three arguments passed to substr[1].

    >perl -MDevel::Peek -e"$_ = 'abcdef'; my $ref = \substr($_, 3, 2); Dum +p($$ref);" SV = PVLV(0x4d3d24) at 0x2cb29c <--- PVLV REFCNT = 1 FLAGS = (TEMP,GMG,SMG) <--- "get" magic and "set" magic IV = 0 NV = 0 PV = 0 MAGIC = 0x49f09c MG_VIRTUAL = &PL_vtbl_substr <--- Function pointers for magic MG_TYPE = PERL_MAGIC_substr(x) <--- x = substr magic TYPE = x <--- x = substr magic TARGOFF = 3 <--- For "x", start offset of substring TARGLEN = 2 <--- For "x", length of substring TARG = 0x4a8414 <--- For "x", addr of associated scalar ($_) FLAGS = 0 SV = PV(0x2c8a6c) at 0x4a8414 <--- Dump of associated scalar ($_) REFCNT = 2 FLAGS = (POK,pPOK) PV = 0x2cab94 "abcdef"\0 CUR = 6 LEN = 12

    When you try to fetch from $$ref, the associated get magic first does something like $$ref = substr($$TARG, $TARGOFF, $TARGLEN);.

    After you assign to $$ref, the associated "set" magic effectively does substr($$TARG, $TARGOFF, $TARGLEN, $$ref);.


    Notes

    1. Missing and negative arguments are resolved before being assigned.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1054793]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (7)
As of 2014-09-16 09:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (158 votes), past polls