How can I create an iterator that works inside foreach()

[PUCKERING]

I am trying to create an iterator that will work inside a foreach (). No problem creating one for a while (), but foreach () eludes me.

I've tried a closure, and a class (with a next method), and even the Iterator module. No joy.

Here's an example, cribbed from the Iterator POD:

use Iterator;

sub irange_limited {
    my ($start, $end) = @_;
 
    return Iterator->new ( 
        sub {
            Iterator::is_done()
                if $start > $end;
     
            return $start++;
        }
    );
}

my $it = irange_limited(3, 5);

foreach ( $it->value ) {say $_ }

That prints 3, when it should print lines containing 3, 4 and 5.

Even the venerable High Order Perl book, which has an entire chapter devoted to iterators and some fine examples, fails to mention anything about using them in foreach -- or not being able to. And while foreach is used within the implementation of some examples, they use cases all use while(). This leads me to think there may be some fundamental reason why it cannot work.

Your wisdom and enlightenment, as always, would be much appreciated.

[AnomalousMonk]

Here's one way. The iterator in this example is very simple: it has no way to reset or reinitialize once it is exhausted, and it will only reach exhaustion in a while-loop or equivalent (update: i.e., in scalar-context invocation).

c:\@Work\Perl>perl

use strict;
use warnings;

# use Data::Dump qw(dd);  # for debug

sub Iterator (&) { return $_[0]; }  # syntactic sugar per mjd (Dominus
+)

sub irange_limited {

    my ($start, $end) = @_;

    return Iterator {
        return if $start > $end;
        return wantarray ? $start .. $end : $start++;
        };

    }

my $iter = irange_limited(3, 5);

for my $n ($iter->()) {
    print "for loop: $n \n";
    }

while (my $n = $iter->()) {
    print "while loop: $n \n";
    }

^Z
for loop: 3
for loop: 4
for loop: 5
while loop: 3
while loop: 4
while loop: 5

Update: Here's a slightly more sophisticated iterator. It always returns the full range when invoked in list context, and it will automatically reset when exhausted upon invocation in scalar context. Note, however, that list and scalar context invocations are independent of each other!

sub irange_limited {

    my ($start, $end) = @_;

    my $n = $start;

    return Iterator {

        return
            wantarray ? $start .. $end    :
            $n > $end ? ($n = $start, ()) :
            $n++
            ;

        };

    }

(Update: Also note that this solution has a semipredicate problem. The range -1, 1 will result in a value of 0 (false) that will terminate a while-loop prematurely. A further defined test is needed in the while-loop condition because undef (also false) flags iterator exhaustion. (Update: There are also other solutions to this problem :))

[PUCKERING]

Brilliant!!! The & prototype was the secret sauce! Thank you so much for your help.

I'm embarrassed to see that it was on page 123 of High Order Perl and I missed it. An example with a foreach would have helped -- but MJD did such an awesome job with a lot of complicated topics in that book that there's no way I'm going to complain! I should have read the chapter more carefully.

There's a web site called programming-idioms.org which provides a database of idioms implemented in different languages for easy comparison. The reason I asked about this is that I made a contribution to the idiom for generator functions but after additional testing realized it didn't work in a foreeach loop. I'll use your suggestion to update it.

Here's a link to the programming idioms page I'll be modifying:

Idiom #319 generator functions

[AnomalousMonk]

The & prototype was the secret sauce!

Actually, I think wantarray is the secret sauce as it allows the iterator function to discriminate list vs. scalar context. :)

[hv]

In the context of that website, it's possible that the intent is something more like the C-style for loop, in which case something like the following might be a better fit:

sub range {
  my($start, $end) = @_;
  return sub {
    return undef if $start > $end;
    return $start++;
  };
}

my $it = range(3, 5);
for (my $value; $value = $it->(); ) { say $value }

(But actual idiomatic perl would use a while loop here.)

If the intent is to handle a list-style for loop, the "iterator" can be made much simpler by handling just that case, which I think is similar to the Ruby solution shown:

sub list_range {
  my($start, $end) = @_;
  return sub { $start .. $end };
}

my $lit = list_range(3, 5);
for ($lit->()) { say $_ }

In the current example perl code at the link, the "_upto" function is not needed, it can simply be replaced with "sub", as in my examples above: the sub keyword in this context yields an anonymous subroutine reference from the block that follows it, and Higher Order Perl should be telling you all about that. The comment "To work in a foreach each loop, inner sub upto must be predeclared to take an anonymous sub argument" is wrong - if we have a subroutine reference, $it->() will happily invoke it, whether it's in a for loop or not.

[AnomalousMonk]

Also, looking at the post on the other site, it occurs to me that you might mention the semipredicate problem with the posted code (see the end of my updated post).

... inner sub upto must be predeclared to take an anonymous sub argument; hence the (%).

I don't understand this. Are upto (no leading underscore) and (%) typos? And shouldn't _upto be something like Iterator (for clarity)?

[hv]

The significant difference between foreach (EXPR) BLOCK and while (EXPR) BLOCK is that foreach evaluates the expression once in list mode getting a single list of results, then invokes BLOCK over the items in that list in turn; on the other hand, while evaluates the expression once (in scalar mode) and immediately invokes BLOCK on that value, then evaluates EXPR again to get another value, repeating until EXPR gives a false value.

The nature of an iterator is that it is designed to be called repeatedly, returning a single result each time. So if you're making an iterator, it won't be designed to work with foreach - you would need instead to have a function that returns a single list of all the values.

So the next question is: why do you want to do this, what problem are you trying to solve?

[jwkrahn]

foreach loops over a LIST. Your iterator returns a list of one value so foreach is satisfied.

[haukex]

One of the major advantages of iterators is that they can produce items one at a time, which can be very significant if the items are expensive to produce and you don't need all of them, or you don't want the upfront cost all at once. Since, as others have said, foreach generally operates on a list, it doesn't seem the most efficient approach to convert an iterator to a list. Consider for example processing the results of HTTP API calls: do you want to first do a ton of HTTP calls, store the results, and then process them, or do you want to process each result as it comes in?

Another thing to consider is that the number of items returned from an iterator may not be known - it may even be infinite!* If you, for example, tie an iterator to an array, using that array in a foreach will cause it to actually iterate over the items returned from the iterator one-by-one, but it will also call FETCHSIZE to find out how many items there are.

So my question would be: why do you want to use an iterator in a foreach instead of while?

(BTW, personally I like Iterator::Simple a little bit better than Iterator. You may also be interested in how I implemented a really simple iterator, including an overloading of the <> operator, in my Algorithm::Odometer::Gray.)

* Update: To preempt any questions about what good an infinite iterator is: they can be used in operations like islice or zip.