Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Is it safe to append to the array you are iterating over

by DrWhy (Chaplain)
on Mar 14, 2013 at 16:07 UTC ( #1023497=perlquestion: print w/ replies, xml ) Need Help??
DrWhy has asked for the wisdom of the Perl Monks concerning the following question:

My Google-fu is failing me today. This seems like a question that should have been answered already somewhere, but I can't find it.

In Perl, is it safe to push new elements onto an array that you are currently looping over. E.g.:

my @arr = qw/a b c/; foreach (@arr) { push @arr, 'd' if $_ eq 'a'; push @arr, 'e' if $_ eq 'b'; push @arr, 'f' if $_ eq 'c'; print $_; } print "\n";
I have tested this and it does work. But I wonder if this practice is safe in general for Perl arrays?

Point of clarification: I am asking about a specific use case that I have found that contradicts what the standard perl docs say about modifying arrays used in loop lists. If 1. the entire loop list is a single array and 2. the only modifications you make to this list inside the loop are appends (push), then the loop iterator recognizes the additional elements without getting 'confused'. The question then is is this violation of the general prohibition safe or was I lucky (e.g., is this behavior an implementation quirk specific to a version of Perl that could change without warning in different perl versions)?

The Perl I used to verify this is 'v5.10.0 built for x86_64-linux-thread-multi'

--DrWhy

"If God had meant for us to think for ourselves he would have given us brains. Oh, wait..."

Comment on Is it safe to append to the array you are iterating over
Download Code
Re: Is it safe to append to the array you are iterating over
by toolic (Chancellor) on Mar 14, 2013 at 16:34 UTC
      Right: "If any part of LIST is an array, foreach will get very confused if you add or remove elements within the loop body, for example with splice. So don't do that"

      Except in my case I found that if all you have is a single array as your loop list and you only modify it by push-ing new elements on to the end, the foreach in fact does not seem to get confused.

      My question is is this use case safe or was I lucky?

      Maybe the phrase 'in general' in the OP was too general. I am only asking about cases where the loop list consists solely of a single array.

      --DrWhy

      "If God had meant for us to think for ourselves he would have given us brains. Oh, wait..."

        I think that when the documentation says "don't do that", and you find that in a specific case it's not causing a problem, you're probably lucky by implementation, and not by guarantee. The behaviour may never change, but could change without notice.


        Dave

Re: Is it safe to append to the array you are iterating over
by SuicideJunkie (Priest) on Mar 14, 2013 at 17:27 UTC

    If you don't mind consuming the array:

    while (@array) { $item = shift @array; ... push @array, @newstuff; }
    or a manual index value to preserve the array contents:
    for (my $i=0; $i< $#array; $i++) { ... push @array, $thing if desired(); }
      I've decided to change my code to instead use a while loop/shift solution similar to what you suggested. It's approximately as simple as my original and doesn't have this issue of possibly depending on an undocumented implementation detail of my version of Perl.

      --DrWhy

      "If God had meant for us to think for ourselves he would have given us brains. Oh, wait..."

        possibly depending on an undocumented implementation detail of my version of Perl.

        You most definitely are. The loop is suppose to take a list over which to iterate, but you're relying on the observable differences between that an the actual optimised implementation. You can see this by changing foreach (@arr) to what should be equivalent foreach ((),@arr).

        Either iterate over the indexes

        my @arr = qw/a b c/; for (my $i=0; $i<@arr; ++$i) { my $e = $arr[$i]; push @arr, 'd' if $e eq 'a'; push @arr, 'e' if $e eq 'b'; push @arr, 'f' if $e eq 'c'; print $e; } print "\n";

        Or use a queue

        my @arr = qw/a b c/; while (@arr) { my $e = shift(@arr); push @arr, 'd' if $e eq 'a'; push @arr, 'e' if $e eq 'b'; push @arr, 'f' if $e eq 'c'; print $e; } print "\n";
Re: Is it safe to append to the array you are iterating over
by BrowserUk (Pope) on Mar 14, 2013 at 17:49 UTC

    You can easily make that safe by substituting a list of indexes for the actual items:

    my @arr = qw/a b c/; foreach ( 0 .. $#arr ) { $_ = $arr[ $_ ]; push @arr, 'd' if $_ eq 'a'; push @arr, 'e' if $_ eq 'b'; push @arr, 'f' if $_ eq 'c'; print $_; } print "\n";

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Is it safe to append to the array you are iterating over
by bart (Canon) on Mar 14, 2013 at 17:53 UTC
    The question then is is this violation of the general prohibition safe or was I lucky
    Now I am confused. What do you expect,
    1. Perl loops over the original array, whatever you do to the array in the meantime
    2. Perl loops over the modified array
    Frankly, I have no idea what to expect, you can make a case for both behaviors. In such a case, I'd say: don't do it. What will happen in this case might change over time, possibly because someone decided to fix perl's "erratic behavior".

    What I think is safe, is this:

    1. Loop over the original array: make a temporary copy.
      foreach(my @temp = @arr) { push @arr, 'd' if $_ eq 'a'; push @arr, 'e' if $_ eq 'b'; push @arr, 'f' if $_ eq 'c'; print $_; }
    2. if you want to loop over the modified array, use the loop index:
      for(my $i = 0; $i < @arr; $i++) { local $_ = $arr[$i]; push @arr, 'd' if $_ eq 'a'; push @arr, 'e' if $_ eq 'b'; push @arr, 'f' if $_ eq 'c'; print $_; }
    p.s. I was expecting this to work:
    foreach(() = @arr) { print; }
    but it doesn't loop even once. I think that's a bit weird, since
    $x = () = @arr;
    actually sets $x to the size of the array.
      My expectation was your number 2. Loop over the modified array. And that's what it does in my experiments. It looks like Perl is being efficient and rather than constructing a new list to loop over that is just a verbatim copy of the array it already has access to, it's using that array as it's loop list and the implementation of arrays in Perl is such that adding elements on the end of the array doesn't mess up the iterator used by the for loop. At least for the small size arrays I've been working with. Thinking about it further I can imagine Perl doing this until the array gets to a large enough size that it can't be held in the current location in memory so it moves it around in a way that could invalidate or reset the iterator.

      --DrWhy

      "If God had meant for us to think for ourselves he would have given us brains. Oh, wait..."

Re: Is it safe to append to the array you are iterating over
by kcott (Abbot) on Mar 14, 2013 at 18:01 UTC

    G'day DrWhy,

    In what sense does that code work? In what sense do you consider it to be safe?

    I don't see any code requesting for warnings to be emitted. I don't see any sanity checks. I don't see any tests for ... well, anything. Perhaps worked safely means it finished before flames burst forth from the back of your computer :-)

    What would happen if, at some later time, a fourth conditional push() was added:

    push @arr, $some_var if $_ eq 'd';

    If $some_var happens to evaluate to 'a', you're now stuck in an infinite loop.

    There's all sorts of other scenarios that result in infinite loops, chewing up inordinate amounts of processor or memory resources, leaving the array in a state that has adverse consequences for subsequent processing and so on.

    In general, this practice is not safe; moreover, it can easily be avoided.

    -- Ken

      By the code working I expect that the for loops over the entire array contents including the elements added in the loop body. The expected output of this code is
      abcdef
      
      Which is in fact what I get when I run it. By safe I mean it doesn't crash or do something else unexpected when I run it. I added use strict and use warnings and the code still worked fine. No warnings emitted, no strictness violations. I'm not sure what kind of sanity checks you might add to this code. I suppose you could check at the end of the loop that the array had exactly the expected six elements in it.

      It is clear to me that this technique used incorrectly could lead to creation of an infinite loop, but that is just a programmer/algorithmic GIGO type issue.

      Yes, you can argue that it is unsafe because a user could accidentally write their algorithm in such a way that some data causes it too loop infinitely. But that's not the kind of safe I'm asking about. By safe I mean this. If I write a piece of code using this pattern and it works as I expected in this version of perl, is it likely to break unexpectedly when I run the same code on a different version of perl, because the fact of it working as expected depended on an implementation detail of the interpreter rather than a supported feature of the interpreter.

      Or to restate the original question. The following behavior works in Perl 5.10.0 (linux/multi threaded): a for loop using an array as the sole element of its loop list can loop over the entire array even if that array is extended inside the loop body -- so the definition of 'entire' changes as the loop is executed and the array gets longer. Is this behavior a supported feature of the language which can be relied upon to work generally across future versions of perl or is it an implementation detail subject to unexpected and undocumented change?

      --DrWhy

      "If God had meant for us to think for ourselves he would have given us brains. Oh, wait..."

        The problem is that even if the logic in the loop body doesn't cause an infinite loop; you are at the very least also iterating over all the additional elements that you push for no good reason. You've just pushed those values; you presumably don't want them to cause further pushes.

        Why do pointless extra work when it is so easily avoided?


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
        /div
Re: Is it safe to append to the array you are iterating over (yes)
by Anonymous Monk on Mar 14, 2013 at 19:47 UTC
      I wouldn't be surprised now if even splice might not confuse the loop as long as you are splicing parts of the array that the loop iterator hasn't reached yet.

      --DrWhy

      "If God had meant for us to think for ourselves he would have given us brains. Oh, wait..."

Re: Is it safe to append to the array you are iterating over
by Discipulus (Curate) on Mar 15, 2013 at 09:00 UTC
    i found this very interesting.. it is incredible after more then 10 years using Perl, discover 'new' point of view about some simple as a  foreach (@array) loop.

    ++ to Re: Is it safe to append to the array you are iterating over and Re: Is it safe to append to the array you are iterating over (yes)

    I'm happy to see that, once again,Perl is doing it correctly: it is a logical operetion append to a list while iterating over it. Other operations are confusing.

    May be worth to add some words to Perldoc: "...get very confused if you add or remove elements within the loop body (unless you merely push something at the end),....".

    L*

    there are no rules, there are no thumbs..
      I would not be in favor of making this documentation change, but since this is effectively adding to the list of supported features of the language, turning an implementation detail into a officially supported behavior, I wouldn't do this without adding tests for it in the perl test suite, and getting approval from whoever gives approval for behavioral changes to the language.

      That is, this change should be treated as not just a 'doc fix' but a new feature.

      --DrWhy

      "If God had meant for us to think for ourselves he would have given us brains. Oh, wait..."

Re: Is it safe to append to the array you are iterating over
by LanX (Canon) on Jul 10, 2013 at 16:03 UTC
    not a good idea!

    DB<104> @arr = qw/a b c/; => ("a", "b", "c") DB<105> foreach ( "x",@arr,"y") { push @arr, 'd' if $_ eq 'a'; push @arr, 'e' if $_ eq 'b'; push @arr, 'f' if $_ eq 'c'; print $_; } => "" xabcy

    better rely on orthogonal behavior!

    Cheers Rolf

    ( addicted to the Perl Programming Language)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1023497]
Approved by toolic
Front-paged by Lotus1
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (8)
As of 2014-10-21 07:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (98 votes), past polls