perltutorial
ikegami
<h1>Operator Associativity and Eliminating Left-Recursion in [mod://Parse::RecDescent]</h1>
<h2>Synopsis</h2>
<p>I have found documentation on eliminating left-recursion (such as [id://153155]) to be unsatisfactory. Left recursion is usually eliminated at the expense of associativity. This tutorial seeks to address this issue.
<p>The document provides two implementations for every topic covered. The first shows how the topic applies when evaluating the text at parse time. The second shows how the topic applies when building a parse tree. It is probably best to ignore the latter (parse tree creation) until the former (parse-time eval) is understood.
<p>Feedback and criticisms are welcome.
<h2>Table of Contents</h2>
<ul>
<li><a href="#n553889_s1">1. What is Operator Associativity?</a>
<li><a href="#n553889_s2">2. Parsers and Associativity</a>
<ul>
<li><a href="#n553889_s2_a">2.a. Parse-time Evaluation with Associativity</a>
<li><a href="#n553889_s2_b">2.b. Building a Parse Tree with Associativity</a>
</ul>
<li><a href="#n553889_s3">3. Eliminating Left-Recursion</a>
<ul>
<li><a href="#n553889_s3_a">3.a. Method 1: Create a Flat List, and Reconstruct</a>
<ul>
<li><a href="#n553889_s3_a_i">3.a.i. ...to Evaluate the Text at Parse-time</a>
<li><a href="#n553889_s3_a_ii">3.a.ii. ...to Build a Parse Tree</a>
</ul>
<li><a href="#n553889_s3_b">3.b. Method 2: Create a Flat List Using <c><leftop></c>, and Reconstruct</a>
<ul>
<li><a href="#n553889_s3_b_i">3.b.i. ...to Evaluate the Text at Parse-time</a>
<li><a href="#n553889_s3_b_ii">3.b.ii. ...to Build a Parse Tree</a>
</ul>
<li><a href="#n553889_s3_c">3.c. Method 3: Using a Subrule Argument</a>
<ul>
<li><a href="#n553889_s3_c_i">3.c.i. ...to Evaluate the Text at Parse-time</a>
<li><a href="#n553889_s3_c_ii">3.c.ii. ...to Build a Parse Tree</a>
</ul>
</ul>
<li><a href="#n553889_s4">4. Improving Right-Recursion</a>
<ul>
<li><a href="#n553889_s4_a">4.a. Method 1: Create a Flat List, and Reconstruct</a>
<ul>
<li><a href="#n553889_s4_a_i">4.a.i. ...to Evaluate the Text at Parse-time</a>
<li><a href="#n553889_s4_a_ii">4.a.ii. ...to Build a Parse Tree</a>
</ul>
<li><a href="#n553889_s4_b">4.b. Method 2: Create a Flat List Using <c><rightop></c>, and Reconstruct</a>
<ul>
<li><a href="#n553889_s4_b_i">4.b.i. ...to Evaluate the Text at Parse-time</a>
<li><a href="#n553889_s4_b_ii">4.b.ii. ...to Build a Parse Tree</a>
</ul>
<li><a href="#n553889_s4_c">4.c. Method 3: Using a Subrule Argument</a>
<ul>
<li><a href="#n553889_s4_c_i">4.c.i. ...to Evaluate the Text at Parse-time</a>
<li><a href="#n553889_s4_c_ii">4.c.ii. ...to Build a Parse Tree</a>
</ul>
</ul>
<li><a href="#n553889_s5">5. Working Code</a>
<ul>
<li><a href="#n553889_s5_a">5.a. ...to Evaluate the Text at Parse-time</a>
<li><a href="#n553889_s5_b">5.b. ...to Build and Evaluate a Parse Tree</a>
</ul>
</ul>
<readmore>
<h2><a name="n553889_s1">1. What is Operator Associativity?</a></h2>
<p>The Perl binary operators <c>+</c> and <c>-</c> have the same precedence, but that doesn't mean they can be evaluated in any order. For example, consider <c>4 - 5 + 6</c>.
<c>
If executed from left-to-right,
4 - 5 + 6 = (4 - 5) + 6 = 5
If executed from right-to-left,
4 - 5 + 6 = 4 - (5 + 6) = -7
</c>
<p>Similarly,
<c>
If executed from left-to-right,
4 ** 3 ** 2 = (4 ** 3) ** 2 = 4096
If executed from right-to-left,
4 ** 3 ** 2 = 4 ** (3 ** 2) = 262144
</c>
<p>Operators which are evaluated from left-to-right are left-associative.
<p>Operators which are evaluated from right-to-left are right-associative.
<p>In Perl, binary operators <c>+</c> and <c>-</c> are left-associative, and binary operator <c>**</c> is right-associative. (Refer to [doc://perlop#Operator-Precedence-and-Associativity--operator%2c-precedence-precedence-associativity|Operator Precedence and Associativity] in [doc://perlop] for the associativity of other operators.)
<h2><a name="n553889_s2">2. Parsers and Associativity</a></h2>
<p>Grammars do not specify associativity. A [wp://Formal grammar|grammar] simply defines whether a given string is valid in the language represented by the grammar, and associativity is not needed for that purpose.
<p>However, we're rarely just interested in validity check. Parsers that return a parse tree representing the text being parsed and those that evaluate the text being parsed are much more useful. Because Parse::RecDescent processes rules from left to right, grammars can be written in a form that lends itself well to doing these tasks.
<p>Left-associative:
<c>
sum : sum /[+-]/ NUM
| NUM
</c>
<p>Right-associative:
<c>
pow : NUM '**' pow
| NUM
</c>
<p>The following subsections will enrich these grammars with code to build a parse tree and to evaluate the expression at parse-time. As you will see, no changes will be needed to the grammar.
<h3><a name="n553889_s2_a">2.a. Parse-time Evaluation with Associativity</a></h3>
<p>Left-associative:
<c>
sum : sum '+' NUM { $item[1] + $item[3] }
| sum '-' NUM { $item[1] - $item[3] }
| NUM { $item[1] }
</c>
<p>Right-associative:
<c>
pow : NUM '**' sum { $item[1] ** $item[3] }
| NUM { $item[1] }
</c>
<h3><a name="n553889_s2_b">2.b. Building a Parse Tree with Associativity</a></h3>
<p>Left-associative:
<c>
sum : sum /[+-]/ NUM { [ @item[2,1,3] ] }
| NUM { [ $item[1] ] }
</c>
<p>Right-associative:
<c>
pow : NUM '**' pow { [ @item[2,1,3] ] }
| NUM { [ $item[1] ] }
</c>
<h2><a name="n553889_s3">3. Eliminating Left-Recursion</a></h2>
<p>There is a catch. The theory is solid, but parsers have limitations.
<p>Productions of the form <c>a : a b</c> are called left-recursive. An entire class of parser generators cannot process left-recursive grammars, and Parse::RecDescent belongs to that class. Unfortunately, the left-associative rules presented so far are left-recursive. The remainder of this section will show methods of removing left-recursion from grammars for Parse::RecDescent.
<h3><a name="n553889_s3_a">3.a. Method 1: Create a Flat List, and Reconstruct</a></h3>
<p>It's easy to parse <c>4 - 5 + 6</c> into the list <c>'4', '-', '5', '+', '6'</c>. The following snippet does so:
<c>
sum : NUM sum_ { [ $item[1], @{$item[2]} ] }
sum_ : /[+-]/ NUM sum_ { [ $item[1], $item[2], @{$item[3]} ] }
| { [] }
</c>
<p>If we are evaluating at parse-time, we have little choice but to process the sum as a list rather than a binary operator. When building a parse tree, we have two options. We could leave it as is, or we could convert the list into a tree.
<p>The following subsections show how to evaluate the list and how to treeify it.
<h4><a name="n553889_s3_a_i">3.a.i. ...to Evaluate the Text at Parse-time</a></h4>
<c>
{
sub eval_sum {
my $acc = shift(@_);
while (@_) {
my $op = shift(@_);
if ($op eq '+') { $acc += shift(@_); }
elsif ($op eq '-') { $acc -= shift(@_); }
}
return $acc;
}
}
sum : NUM sum_ { eval_sum($item[1], @{$item[2]}) }
sum_ : /[+-]/ NUM sum_ { [ $item[1], $item[2], @{$item[3]} ] }
| { [] }
</c>
<h4><a name="n553889_s3_a_ii">3.a.ii. ...to Build a Parse Tree</a></h4>
<c>
{
sub treeify {
my $t = shift(@_);
$t = [ shift(@_), $t, shift(@_) ]
while @_;
return $t;
}
}
sum : NUM sum_ { treeify($item[1], @{$item[2]}) }
sum_ : /[+-]/ NUM sum_ { [ $item[1], $item[2], @{$item[3]} ] }
| { [] }
</c>
<h3><a name="n553889_s3_b">3.b. Method 2: Create a Flat List Using <c><leftop></c>, and Reconstruct</a></h3>
<p>This method is the same as Method 1, but takes advantage of a Parse::RecDescent feature to improve readability. Parse::RecDescent has a pair of directives to help build lists. <c><leftop></c> is designed to build left-associative lists, and <c><rightop></c> is designed to build right-associative lists.
<h4><a name="n553889_s3_b_i">3.b.i. ...to Evaluate the Text at Parse-time</a></h4>
<c>
{
sub eval_sum {
my $acc = shift(@_);
while (@_) {
my $op = shift(@_);
if ($op eq '+') { $acc += shift(@_); }
elsif ($op eq '-') { $acc -= shift(@_); }
}
return $acc;
}
}
sum : <leftop: NUM /[+-]/ NUM> { eval_sum(@{$item[1]}) }
</c>
<h4><a name="n553889_s3_b_ii">3.b.ii. ...to Build a Parse Tree</a></h4>
<c>
{
sub treeify {
my $t = shift(@_);
$t = [ shift(@_), $t, shift(@_) ]
while @_;
return $t;
}
}
sum : <leftop: NUM /[+-]/ NUM> { treeify(@{$item[1]}) }
</c>
<h3><a name="n553889_s3_c">3.c. Method 3: Using a Subrule Argument</a></h3>
<p>Normally, information passes from subrule to superrule. For example, in the following code, rule2 receives the result of rule3. In turn, rule1 receives the result of rule2.
<c>
rule1: token rule2
rule2: token rule3
rule3: token
</c>
<p>The deeper something is, the sooner it will get executed. In a list, that means the last (right-most) element encountered will be executed first. With left-associative lists, the opposite is needed. With left-associative lists, information needs to flow from the superrule to the subrule. Fortunately, Parse::RecDescent provides a means of passing information to subrules: Subrule argument lists.
<p>Think of each rule as a function, and of each reference to that rule as a function call. (In fact, this is how the compiled grammars are implemented.) Just like functions can have arguments, so can subrules.
<h4><a name="n553889_s3_c_i">3.c.i. ...to Evaluate the Text at Parse-time</a></h4>
<c>
sum : NUM sum_[ $item[1] ]
sum_ : '+' NUM sum_[ $arg[0] + $item[2] ]
| '-' NUM sum_[ $arg[0] - $item[2] ]
| { $arg[0] }
</c>
<h4><a name="n553889_s3_c_ii">3.c.ii. ...to Build a Parse Tree</a></h4>
<c>
sum : NUM sum_[ $item[1] ]
sum_ : '+' NUM sum_[ [ $item[1], $arg[0], $item[2] ] ]
| '-' NUM sum_[ [ $item[1], $arg[0], $item[2] ] ]
| { $arg[0] }
</c>
<h2><a name="n553889_s4">4. Improving Right-Recursion</a></h2>
<p>Earlier, we ended up with the following rules for right-recursive binary operators:
<c>
pow : NUM '**' pow
| NUM
</c>
<p>Unlike left-recursion, Parse::RecDescent has no problem with right-recursion. However, Parse::RecDescent handles rules with productions with identical prefixes very inefficiently.
<p>Just like in algebra, we can factor out the common prefix into another rule.
<c>
pow : NUM pow_
pow_ : '**' pow
|
</c>
<p>The complicated part is how to evaluate the expression or build the parse tree when one of the operands is matched by one rule, and the other is matched by a different rule. It turns out that doing this is very similar to eliminating left-recursion.
<h3><a name="n553889_s4_a">4.a. Method 1: Create a Flat List, and Reconstruct</a></h3>
<p>Just like when eliminating left-recursion, we can build a flat list of the whole chain of powers, and work with that. The difference is that the list will be processed from right to left.
<h4><a name="n553889_s4_a_i">4.a.i. ...to Evaluate the Text at Parse-time</a></h4>
<c>
{
sub eval_pow {
my $acc = pop(@_);
while (@_) {
my $op = pop(@_);
$acc = pop(@_) ** $acc;
}
return $acc;
}
}
pow : NUM pow_ { eval_pow($item[1], @{$item[2]}) }
pow_ : '**' NUM pow_ { [ $item[1], $item[2], @{$item[3]} ] }
| { [] }
</c>
<h4><a name="n553889_s4_a_ii">4.a.ii. ...to Build a Parse Tree</a></h4>
<c>
{
sub treeify_r {
my $t = pop;
$t = [ pop, pop, $t ]
while @_;
return $t;
}
}
pow : NUM pow_ { treeify_r($item[1], @{$item[2]}) }
pow_ : '**' NUM pow_ { [ $item[1], $item[2], @{$item[3]} ] }
| { [] }
</c>
<h3><a name="n553889_s4_b">4.b. Method 2: Create a Flat List Using <c><rightop></c>, and Reconstruct</a></h3>
<p>Just like Parse::RecDescent has a directive for creating a flat list for a left-associative operator (<c><leftop></c>), it has one to create a flat list for a right-associative operator (<c><rightop></c>).
<h4><a name="n553889_s4_b_i">4.b.i. ...to Evaluate the Text at Parse-time</a></h4>
<c>
{
sub eval_pow {
my $acc = pop(@_);
while (@_) {
my $op = pop(@_);
$acc = pop(@_) ** $acc;
}
return $acc;
}
}
pow : <rightop: NUM /(\*\*)/ NUM> { eval_pow(@{$item[1]}) }
</c>
<h4><a name="n553889_s4_b_ii">4.b.ii. ...to Build a Parse Tree</a></h4>
<c>
{
sub treeify_r {
my $t = pop;
$t = [ pop, pop, $t ]
while @_;
return $t;
}
}
pow : <rightop: NUM /(\*\*)/ NUM> { treeify_r(@{$item[1]}) }
</c>
<h3><a name="n553889_s4_c">4.c. Method 3: Using a Subrule Argument</a></h3>
<p>Let's look at the algebra again. We can change
<c>
pow : NUM '**' pow { $item[1] ** $item[3] }
| NUM { $item[1] }
</c>
<p>into
<c>
pow : NUM pow_
pow_ : '**' pow { <<pow's $item[1]>> ** $item[2] }
| { <<pow's $item[1]>> }
</c>
<p>The problem is that we have to pass <c>$item[1]</c> from <c>pow</c> to <c>pow_</c>. We've already seen that we can pass data from one rule to another using subrule arguments. When eliminating left-recursion, we used the subrule argument to form a stack. When improving right-recursion, we simply pass from the main rule to the helper rule.
<h4><a name="n553889_s4_c_i">4.c.i. ...to Evaluate the Text at Parse-time</a></h4>
<c>
pow : NUM pow_[ $item[1] ]
pow_ : '**' pow { $arg[0] ** $item[2] }
| { $arg[0] }
</c>
<h4><a name="n553889_s4_c_ii">4.c.ii. ...to Build a Parse Tree</a></h4>
<c>
pow : NUM pow_[ $item[1] ]
pow_ : '**' pow { [ $item[1], $arg[0], $item[2] ] }
| { $arg[0] }
</c>
<h2><a name="n553889_s5">5. Working Code</a></h2>
<p>The following subsections contain complete, working code to parse expressions formed of the <c>+</c>, <c>-</c> and <c>**</c> binary operators using the Subrule Argument methods. Parentheses are also supported to produce more meaningful results.
<p>In order to support parentheses and to give the operators their proper precedence, the rules used in the upcoming code are slightly different from those seen earlier. Where <c>NUM</c> used to be in the productions, you will now find <c>term</c> (in <c>sum</c>/<c>sum_</c>) and <c>sum</c> (in <c>pow</c>/<c>pow_</c>).
<p>The code of both subsections produce the same output, an uncommented version of the following:
<c>
Demonstrates left-associativity
4-5+6 = 5 got 5
(4-5)+6 = 5 got 5
4-(5+6) = -7 got -7
Demonstrates right-associativity
4**3**2 = 262144 got 262144
(4**3)**2 = 4096 got 4096
4**(3**2) = 262144 got 262144
</c>
<h3><a name="n553889_s5_a">5.a. ...to Evaluate the Text at Parse-time</a></h3>
<c>
use strict;
use warnings;
use Parse::RecDescent ();
my $grammar = <<'__END_OF_GRAMMAR__';
{
use strict;
use warnings;
}
parse : expr /^\Z/ { $item[1] }
# Just an alias
expr : pow
# vvv lowest precedence
# pow : sum '**' pow
# | sum
pow : sum pow_[ $item[1] ]
pow_ : '**' pow { $arg[0] ** $item[2] }
| { $arg[0] }
# sum : sum /[+-]/ term
# | term
sum : term sum_[ $item[1] ]
sum_ : '+' term sum_[ $arg[0] + $item[2] ]
| '-' term sum_[ $arg[0] - $item[2] ]
| { $arg[0] }
# ^^^ highest precedence
term : '(' expr ')' { $item[2] }
| /\d+/
__END_OF_GRAMMAR__
my $parser = Parse::RecDescent->new($grammar)
or die("Bad grammar\n");
foreach my $expr (
'4-5+6', # Demonstrates left-associativity
'(4-5)+6',
'4-(5+6)',
'4**3**2', # Demonstrates right-associativity
'(4**3)**2',
'4**(3**2)',
) {
my $expected = eval $expr;
my $got = $parser->parse($expr);
print("$expr = $expected got $got\n");
}
</c>
<h3><a name="n553889_s5_b">5.b. ...to Build and Evaluate a Parse Tree</a></h3>
<c>
use strict;
use warnings;
use Parse::RecDescent ();
my $grammar = <<'__END_OF_GRAMMAR__';
{
use strict;
use warnings;
}
parse : expr /^\Z/ { $item[1] }
# Just an alias
expr : pow
# vvv lowest precedence
# pow : sum '**' pow
# | sum
pow : sum pow_[ $item[1] ]
pow_ : '**' pow { [ $item[1], $arg[0], $item[2] ] }
| { $arg[0] }
# sum : sum /[+-]/ term
# | term
sum : term sum_[ $item[1] ]
sum_ : /[+-]/ term sum_[ [ $item[1], $arg[0], $item[2] ] ]
| { $arg[0] }
# ^^^ highest precedence
term : '(' expr ')' { $item[2] }
| /\d+/ { [ @item ] }
__END_OF_GRAMMAR__
my $parser = Parse::RecDescent->new($grammar)
or die("Bad grammar\n");
my %eval = (
term => sub { $_[1] },
'+' => sub { eval_node($_[1]) + eval_node($_[2]) },
'-' => sub { eval_node($_[1]) - eval_node($_[2]) },
'**' => sub { eval_node($_[1]) ** eval_node($_[2]) },
);
sub eval_node {
my ($node) = @_;
$eval{$node->[0]}->(@$node);
}
foreach my $expr (
'4-5+6', # Demonstrates left-associativity
'(4-5)+6',
'4-(5+6)',
'4**3**2', # Demonstrates right-associativity
'(4**3)**2',
'4**(3**2)',
) {
my $expected = eval $expr;
my $tree = $parser->parse($expr);
my $got = eval_node($tree);
print("$expr = $expected got $got\n");
}
</c>
</readmore>
<p><b>Update Aug 13, 2006</b>: The examples have been simplified. A right-associative operator is used for the right-associative examples. Parse-time eval was placed before parse tree building. Added section on simplifying right-recursion. Small additions were made here and there to improve clarity. It still needs to link to a tutorial on precedence.
<p><b>Update Jun 13, 2014</b>: Fixed spelling and grammar mistakes identified by hexcoder.
<p><b>Update Oct 3, 2016</b>: Fixed indexing problem raised by an anonymous monk.