http://www.perlmonks.org?node_id=732286

Dear fellow monks,

recently, perhaps due to some slight change in my programming, I have frequently found myself debugging mysterious bugs where, after three or four careful readings of the source code, there are seemingly none. Most of these have boiled down to misunderstanding precedence rules. Example:

sub done { my $self = shift; return not $self->foo and not $self->bar; # gotcha! }

Due to the low precedence of and, the Perl compiler parses this as

(return not $self->foo) and not $self->bar;

which will always return not $self->foo and never evaluate not $self->bar, let alone perform the and. Using the alternative logical operators fixes everything:

sub done { my $self = shift; return !$self->foo && !$self->bar; }

Now, the expression is parsed as return ( (!$self->foo) && (!self->bar) ), which is what I mean. Further surprises of the similar kind include the following line:

my $bool = 1 and 0; # $bool == 1 and $_ == 0; may not be what you expect!

What is your way of avoiding mistakes such as this? Do you avoid using and et al. unless inside parenthesis or in if-else? Some other method of discipline? Peer review? Linting? Agile methods?

--
say "Just Another Perl Hacker";

Replies are listed 'Best First'.
Re: Burned by precedence rules
by toolic (Bishop) on Dec 23, 2008 at 13:44 UTC
    I code in enough languages that it is impractical for me to remember operator precedence in a single language, let alone 5 languages. So, I got into the habit of always using parentheses around expressions. I find that using parentheses more clearly groups the code and makes it easier to read, with the byproduct of avoiding precedence issues.

    I also prefer using Perl's named operators (and, or, etc.) over it's symbolic ones because I think it makes the code easier to read and understand.

      I frequently have this argument with junior programmers.. like you I have coded in enough languages in my time to use parenthesis almost always. It frustrates some people but there can be no confusion to myself or the compiler of the intended consequences.

      I always code defensively because I don't want to have to deal with bugs that I've created.

      You wrote: "I also prefer using Perl's named operators (and, or, etc.) over it's symbolic ones". I personally prefer the symbolic ones because I've been reading C from a young age and they are very natural to me (just as "x" "+" is as natural as saying "multiplied by" or "plus"). And Perl named operators bind low which can frequently cause the very trouble illustrated by this thread. As an aside I came across a junior developer in our organisation who made the very same precedence error (low precedence "and") a few months ago, and this same junior developer had the audacity to deride my C programming background assuming that I couldn't possibly program in Perl as a result..

Re: Burned by precedence rules
by jplindstrom (Monsignor) on Dec 23, 2008 at 16:20 UTC

    My rule of thumb is to always use && and friends in expressions, and "and" and friends for flow control.

    That works well with the precedence rules, and it makes the intention explicit as to what's going on.

    But I also have a rule of thumb to use parens where that makes things more clear.

    /J

Re: Burned by precedence rules
by JavaFan (Canon) on Dec 23, 2008 at 14:15 UTC
    I use all of and, &&, or, ||, ! and not. I don't find it hard to remember that the old, punctuation ones are "high" precedence (just as in C), and the newer English ones are "low" precedence.

    Given two operators, most of the time I know which one has a higher precedence, certainly for operator combinations I use more frequently. But sometimes, I'm not sure. If I'm not sure, I use parenthesis. Or sometimes, I use -MO=Deparse,-p to find out.

Re: Burned by precedence rules
by MidLifeXis (Monsignor) on Dec 23, 2008 at 15:42 UTC

    I never use and, or, and not for putting together "logical expressions"1 (!$b || $c && $d), but always use the higher precedence C-style operators. The only time I use the three letter ones is when "englishing" an expression do {expression} or die. If it is a formula, use the C-style operators, if it is English, use the words.

    I probably made that as clear as mud.

    1 Yeah, I know both of these are logical expressions, I could not find the English words with enough meaning and distinction to clarify the difference.

    Update: What jplindstrom said in the next subthread is spot on to what I tried to say here.

    --MidLifeXis

Re: Burned by precedence rules
by Herkum (Parson) on Dec 23, 2008 at 15:37 UTC

    I like to use and and not in my code, but as JavaFan pointed out, they have a lower precedence. So I find it best to address this by taking the same approach. I always use a if operator and usually have at least 2 return calls. I find it is clearer what I intended to return as a valid value.

    sub done { my $self = shift; return 1 if not $self->foo and not $self->bar; return; }

      Many would find that better expressed as:

      sub done { my $self = shift; return 1 unless $self->foo or $self->bar; return; }

      because it avoids the two negations which confuse the meaning of the and.

      I used to avoid using unless, partly because none of the other languages I've used have had such a thing, but where it drops out a layer of negation I find it helps clean up intent quite nicely.


      Perl's payment curve coincides with its learning curve.
        I disagree strongly with many people then.

        When we are debugging we do not reliably do De Morgan's laws in our head. I heard this a long time ago and did not believe it until one particularly frustrating debugging session. Ever since then I've avoided using unless with any kind of logic at all no matter how nicely it reads while I'm coding.

        I think the precedence thing has been covered, but since DeMorgan's and "unless" got mentioned, a few thoughts about:

        1. reducing Not's - this is a good idea!
        2. DeMorgan's and using pictures instead of equations
        3. Table driven approaches

        I agree that factoring out negations improves readability. If you get too many "not's" in there it can get confusing.
        Without the "unless", this is another way:

        sub done { my $self = shift; return !($self->foo or $self->bar); }

        Read as: I'm not done if I've got some foo or bar. To me this is easier than I'm done if I don't have this and I don't have that, but situations vary.

        As far as converting between "AND's" and "OR's", I always draw a picture, not a table or an equation - just some little doodle on a piece of scratch paper.

        When looking at (!$self->foo && !$self->bar), I would draw an AND gate with two bubbles on the inputs, interpreted as both low's on input means a high output, then by inspection see that this means that any high input yields a low output. Well, that is a NOR gate. So I would draw an OR gate with a bubble on the output. Then I would compare pictures and verify that they really do the same thing.

        So instead of working with some De Morgan's equation, work with the picture. Ask a hardware guy to show you how they do it. 30 minutes learning how to draw and interpret these pictures (simple logic circuit diagrams) will save you a lot of grief.

        If things get too hairy to draw a simple picture, then simplify things. Maybe you combine some things together in an intermediate expression which is used in the following statement, or something like that...

        I guess I'm getting far afield, but there is another technique when things get REALLY hairy, the table driven approach. This is something to consider with "monster if's"... Depending upon the situation, it might or might not work out to be easier to understand and maintain.

Re: Burned by precedence rules
by tilly (Archbishop) on Dec 24, 2008 at 02:03 UTC
    I am at the opposite end of the spectrum from you. I use the English forms and expect low precedence. Since I write as much or more SQL than Perl, and SQL uses the same operators with the same precedence, this works out quite well.

    If I want high precedence I use parens.

    Incidentally your comment about the my $bool = 1 and 0; suggests that you still do not understand what it does. It sets $bool and does nothing else. It will also generate a warning which may be more confusing than helpful.

    As for catching these bugs, unit tests are a very useful tool assuming that your tests document what your code should do, rather than what they do do. (There should be a special hell for people who track down every boundary case where their code does something it shouldn't do and creates tests for those cases just in case someone should consider fixing the code to do what it should have in the first place.)

Re: Burned by precedence rules
by ikegami (Patriarch) on Dec 23, 2008 at 19:39 UTC

    Do you avoid using and et al. unless inside parenthesis or in if-else?

    Yes. I only used them when followed by next, last, return, die or some other flow control expression.

    I wouldn't use them in expressions even if they had the same precedence as they non-text equivalents because visually distinguishing operators from surrounding identifiers is a good thing.

Re: Burned by precedence rules
by swampyankee (Parson) on Dec 24, 2008 at 04:26 UTC

    Like toolic, I work on the principle of "when in doubt, parenthesize." I've also dealt with optimizing compilers that would handle something like

    do i = 1, 10 do j = 1, 10 do k = 1, 10 l = i + j + k enddo enddo enddo

    differently from

    do i = 1, 10 do j = 1, 10 do k = 1, 10 l = (i + j) + k enddo enddo enddo

    in that it would not recognize i + j as constant in the innermost loop in the first fragment, but would in the second. There are also cases where I've needed to force a specific order of operations to preclude under or overflow.

    I also won't rule out the possibility that, unlike the compiler, I can get confused about logical tests that have multiple and's, or's, and not's; using (perhaps superfluous) parentheses will reduce my confusion.


    Information about American English usage here and here. Floating point issues? Please read this before posting. — emc

Re: Burned by precedence rules
by spiritway (Vicar) on Dec 24, 2008 at 05:19 UTC
    My motto is "Even when *not* in doubt, parenthesize". There are just too many subtle ways to get caught by unexpected behavior. Some of these bugs are hard to see, because of *course* that expression evaluates this way - what other way could it be? When Perl disagrees, i am often seriously frustrated trying to track these errors down.
Re: Burned by precedence rules
by lostjimmy (Chaplain) on Dec 23, 2008 at 19:21 UTC
    I completely agree with MidLifeXis and jplindstrom. I always use && and || for expressions, since I learned to program in C, and and and or for flow control. I find that it looks confusing when I come across a statement such as if (x == 1 and y == 2), whereas open ... or die always seems very natural.
Re: Burned by precedence rules
by andreas1234567 (Vicar) on Dec 24, 2008 at 14:12 UTC
    What is your way of avoiding mistakes such as this?
    Test, test and test. Then finally check which branches were left untested by the test suite.
    --
    No matter how great and destructive your problems may seem now, remember, you've probably only seen the tip of them. [1]
Re: Burned by precedence rules
by webfiend (Vicar) on Dec 26, 2008 at 17:15 UTC

    I like to write the dumbest and most readable code that will the job done, and only expand from there when there is a real need for it in the design.

    sub done { my $self = shift; if ( $self->foo or $self->bar ) { return 0; } return 1; }

    I've had many issues bite me in the tail, but precedence is rarely one of them. Then again, I have such a huge itch about readability that I had to fight off the urge to define TRUE and FALSE constants for this tiny slice of code. So take these thoughts with however many grains of salt you desire.

      I would urge you not to define constants for TRUE and FALSE when the language doesn't provide them.

      One of the nastiest bugs I ever had to track down was caused by someone defining these constants slightly differently than the way the language did. We were constantly surprised when !$is_bad, FALSE == $is_bad, and TRUE != $is_bad did not give the same answers.

      My favorite readability move for code like this would be to rename the methods (maybe is_done) to work better in conditionals.

      In your example, I have a bigger problem understanding why foo returning a true value means that we are not done. If, on the other hand, the method were named is_foo_running it would be easier to understand.

      Although I have fought the no magic literals fight for years, true and false are not where I prefer to fight.

      G. Wade

        I disagree.

        The problem is not in defining 'true' and 'false' constants other than ones predefined by the language. The real problem is using any such Boolean constants in (in)equality expressions.

        One should never write anything like "== false" and one should rewrite any such code that one runs into (when practical). For one thing, it is needless complex, even redundant.

        Yes, defining 'true' and 'false' constants exacerbates this problem and the problem is somewhat mitigated when a language has a first-class Boolean data type (that can't be made to hold other than the two special values) -- which also means that the language likely defines its own symbolic representations for those two special values.

        But one can certainly sanely use 'true' and 'false' constants even in Perl. Just don't test for equality against Boolean values, especially against Boolean constants. And one should follow this practice even in languages where a real Boolean datatype mitigates the seriousness of such redundant constructs.

        - tye        

        I had exactly the same thought on the sub name and "false means true" issues, but thought I'd leave it alone for the moment. I do think the use of constants in one form or another is a valid approach, but you need to make sure it's used consistently throughout the entire project. That can be a challenge.

Re: Burned by precedence rules
by sundialsvc4 (Abbot) on Dec 30, 2008 at 22:17 UTC

    “((I) parenthesize (like (crazy))) and (have (never (regretted it))),” he said bluntly, compulsively counting the left and right parentheses in his statement to be sure that they were evenly matched. “(They (cost me) nothing) and (remove (all doubt)).”

    What matters ... truly the only thing that really matters ... is clarity. Anyone and everyone who looks at a line of source-code needs to be able to know, without a shred of doubt or ambiguity, what that line of source will mean to the computer. A few well-placed parentheses, as well as newlines and white-space, will help tremendously in comprehension by the humans (including yourself) who will one day “parse” the code. The cost of any mis-understanding can be huge... both in time and in real money.

    I vividly remember the “programming puzzlers” that used to be put out in magazines like Dr. Dobb's by the purveyors of a certain debugging system. The objective was to read a line of cryptic code and to interpret what it actually meant. I found myself thinking, “gee, I wouldn't want to have to write 50,000 lines of real production software in that language!” (Of course, “that language” was C, and therefore of course this was what I did do for a living, but it really rammed-home the importance of simplicity and clarity.) The computer does not care, but humans (like you) do.

      “((I) parenthesize (like (crazy))) and (have (never (regretted it))),” ... “(They (cost me) nothing) and (remove (all doubt)).”

      If you think that having that number of parens in a single compound condition achieves clarity, may I suggest you learn lisp. You'll be right at home.

      Otherwise, there are far better ways to achieve clarity. From using whitespace, to nested ifs, to breaking sub terms out as functions with meaningful names.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      ( Oops, I thought that was in reply to Re^5: Burned by precedence rules )

      That's exactly my point. Those who usually treat "and" and "or" as flow control statements don't do it because they must. Like you said, the computer doesn't care. To them, it increases the clarity/readability/maintainability of the code.

Re: Burned by precedence rules
by Zarchne (Novice) on Dec 30, 2008 at 04:42 UTC
    Joining with those who only use the English logical operators when then intend flow control, which follows what the perlop man page — “Operator Precedence and Associativity” — recommends, I suggest one also meditate on this:
    &&= and ||= exist; and= and or= do not.
      I suggest one also meditate on this: &&= and ||= exist; and= and or= do not.

      S'funny, but that is exactly what makes them (and & or) so useful in boolean conditions.

      But you will not see that usefulness until you cease making an wholly artificial distinction beween "flow control" and whatever you consider to be "non-flow control".

      This: my $x = isSomething() and isSomethingElse() ? $p : $q;

      is just as much "flow control" as doSomthing() or doSomethingElse();

      as is: if( isSomething() and isSomethingElse() ) { ... } else { ... };


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        You highlighted "wholly artificial" as if it matters, as if a programming concept could be anything else? It's a funny argument to make.