Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Tokenizing and qr// <=> /g interplay

by skyknight (Hermit)
on Apr 23, 2005 at 15:01 UTC ( [id://450720]=perlquestion: print w/replies, xml ) Need Help??

skyknight has asked for the wisdom of the Perl Monks concerning the following question:

I wish to tokenize a stream and I'm interested in understanding the qr// operator a bit better, in particular how one uses the /g modifier with it. I'm not so much interested in pointers to CPAN for modules to do this for me, though I would appreciate pointers on how tokenizing is done if I am going about it in a stupid way.

Specifically, I want to read in a list of possible tokens, build a pattern on the fly to recognize them, and then rip through an input string iteratively, reporting the tokens found within it. I figured that the way to do this was to concatenate all the possible token descriptors with the regex alternation symbol (|), embed the whole thing in parentheses to do capture, get a regex for it using the qr// operator with the /g modifier, and then have a while loop in whose conditional clause I bind the regex to the token stream. So, pseudo-code something like this...

my $stream = 't1 t2 t1 t3 t1 t4'; my @tokens = ('t1, 't2', 't3', 't4'); my $pattern = join('|', @tokens); my $regex = qr/($pattern)/g; while ($stream =~ $regex) { print "token: $1\n"; }

Alas, this seems not to work. Perl gripes "bareword found where operator expected" where I try to put the /g modifier. I wondered if I had botched the syntax, so I tried the /i modifier, and that works, so no, apparently it just doesn't like having the /g modifier stuck on the qr// operator. Hm... So, I changed the while loop to this...

while ($stream =~ /$regex/g) { print "token: $1\n"; }

That works just fine, but I'm left wondering what exactly Perl is doing when I write my code thusly. Namely, I'm wondering if it is any better than doing this...

while ($stream =~ /($pattern)/g) { print "token: $1\n"; }

I want to build this tokenizer pattern only once, and I'm concerned that the ways I've hacked around qr//'s apparent refusal to allow a /g modifier doesn't accomplish this. Namely, I'm worrying that the pattern is getting recompiled every time I use it since I'm embedding it in a new //.

So, what's the deal? Why won't qr// take /g? Is this a bug, a feature, or just a complete lack of comprehension on my part? I'm rather confused.

Replies are listed 'Best First'.
Re: Tokenizing and qr// <=> /g interplay
by Tanktalus (Canon) on Apr 23, 2005 at 15:18 UTC

    For starters, perlop says that the qr operator looks like this: qr/STRING/imosx - g isn't allowed.

    Using $stream =~ /$regex/g still only compiles the regex once as the way that perl interpolates the $regex into a match operator should handle this for free (as in CPU time).

      OK, that seems to make sense... So presumably the following idiom is the right way to go about things...
      while ($stream =~ /$regex/g) { print "token: $1\n"; }
Re: Tokenizing and qr// <=> /g interplay
by japhy (Canon) on Apr 23, 2005 at 15:21 UTC
    The /g modifier is not part of a regex, it's part of the pattern matching operation. The only flags that affect a regex are /i, /m, /s, and /x. In addition, because the contents of qr// are interpolated, the /o flag can be used with the qr// operator.

    A compiled regex (qr//) does not take the place of the pattern matching operation. It merely holds the regex.

    If you want to know when qr// is useful, that's another question that I'll answer (if you ask).

    Jeff japhy Pinyan, P.L., P.M., P.O.D, X.S.: Perl, regex, and perl hacker
    How can we ever be the sold short or the cheated, we who for every service have long ago been overpaid? ~~ Meister Eckhart
      OK, consider it asked. I presume that the most useful thing about qr// is that it allows you to pass regular expressions around as arguments to functions and such, but honestly my lack of experience with it leaves me wondering.
        When you have variables in a regex, Perl examines the contents of those variables to see if the overall representation of the regex has changed:
        for ("ab", "cd") { if ($str =~ /$_/) { ... } }
        In the above code, the regex 'ab' is compiled and executed, and then the regex 'cd' is compiled and executed. Compare that with:
        ($x, $y) = ("ab", "c"); for (1, 2) { if ($str =~ /$x$y/) { ... } ($x, $y) = ("a", "bc"); }
        Here, even though $x and $y change, the ACTUAL regex ('abc') does not change, so the regex is compiled only once. The process that Perl does internally is this:
        1. take regex at this opcode
        2. interpolate any variables
        3. compare with previous value of the regex at this opcode
        4. compile if different
        5. execute this regex
        When you use the /o modifier, it tells Perl that after it has compiled the regex, it should SKIP steps 2-4 of this process, meaning that the regex at this opcode will NEVER change.

        So what is qr// good for? Consider this:

        my @strings = make_10_strings(); for (@strings) { for my $p ('x+', 'yz?y', 'xz+y') { if ($_ =~ $p) { handle($_) } } }
        This code compiles a grand total of 30 regexes. Why? Because for each string in @strings we've got three patterns to execute, and because each time the $_ =~ $p is encountered the contents of $p has changed, the regex is compared and recompiled each time. Now sure, you could reverse the order of the loops, but that will result in the calls to handle() happening in a different order.

        So enter the qr//.

        When Perl sees a regex comprised solely of a single variable, Perl checks to see if that variable is a Regexp object (what qr// returns). If it is, Perl knows that the regex has already been compiled, so it simply uses the compiled regex in the place of the regex. That means doing:

        my @strings = make_10_strings(); for (@strings) { for my $p (qr/x+/, qr/yz?y/, qr/xz+y/) { if ($_ =~ $p) { handle($_) } } }
        is considerably faster. There is no additional compilation happening. It's probably even better to move the qr// values into an array, but that might be moot since they're made of constant strings in this example. The point is, the use of qr// in a looping construct is the primary benefit it offers. Yes, it helps break a regex up into pieces too, but that's just a matter of convenience.

        Be warned that the benefit of qr// objects is lost if there is additional text in the pattern match. I mean that $foo =~ /^$rx_obj$/ suffers from the same problem as $foo =~ /^$text$/.

        Jeff japhy Pinyan, P.L., P.M., P.O.D, X.S.: Perl, regex, and perl hacker
        How can we ever be the sold short or the cheated, we who for every service have long ago been overpaid? ~~ Meister Eckhart
        Another use for qr// is to break up unmanageably complex regular expressions into simpler, named, self-contained pieces. (There's a direct parallel here with subs, which do the same for 'ordinary' Perl code. In fact, you can consider a named regex to be just a function written with a funny-looking syntax: its input is a string and its output is either a Boolean value or one or more strings, depending on whether it captures anything.)

        Here's an example from a code-filtering assertions module (yes, another one) that's not yet tested thoroughly enough to submit to CPAN:

Re: Tokenizing and qr// <=> /g interplay
by tlm (Prior) on Apr 23, 2005 at 15:25 UTC

    The full spec for qr in perlop is qr/STRING/imosx, so what you observed is the expected behavior.

    the lowliest monk

      Hm. I was looking in the Camel book and I didn't see that. I'll have to remember to look at the man pages as well in the future.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://450720]
Approved by ktross
Front-paged by tlm
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (3)
As of 2024-05-24 05:14 GMT
Find Nodes?
    Voting Booth?

    No recent polls found