Return a list or an iterator

[nferraz]

I don't know if this is a common idiom, but I thought it was interesting, so I'm sharing with you.

This is a common way to return lists or references, depending on the calling context:

    sub foo {
        # ...
        return wantarray? @foo : \@foo;
    }

Now, since I was studying how to create iterators using closures, I tried this:

    sub foo {
        # ...
        return wantarray? @foo : sub { shift @foo if @foo };
    }

In other words, "return a list or an iterator": we can call the function like this:

    my @array = foo( $x );

    foreach ( @array ) {
        # do something
    }

And we can call it "iteractivelly":

    my $iter = foo( $x );

    while ( my $row = $iter->() ) {
        # do something
    }

As I said, I'm not sure if this is a common idiom (or even an useful one), but it seemed interesting so I decided to share with you. : )

[ysth]

That is interesting. One change I'd suggest would be to not say if @foo, since that will result in the iterator returning 0 (@foo in scalar context) when the list is exhausted. Unconditionally saying shift @foo will return undef instead, which, while still ambiguous if @foo may contain undef, is more what I'd expect.

Either way, your iterator is only expecting to be called in scalar context; you could expand it to allow returning the remaining list when the iterator is called in list context:

sub foo {
    # ...
    wantarray ? @foo : sub { wantarray ? splice @foo : shift @foo } };
}

(It is a little inconsistent to use return in sub foo but not in the anonymous sub.)

[Anonymous Monk]

The safe way to handle undefs as a valid element is to have the iterator return one element in list context and an empty list if there's no elements, and then always call it in list context when iterating over it.

my $iter = sub { @foo ? shift @foo : () };

while (my ($elem) = $iter->()) {
    # $elem may be undefined.
}

This way you don't need it to be an object and you don't need any special end-of-list elements.

[gaal]

Well, that's a typical semipredicate problem with standard ways to be treated. If you know the kinds of data foo produces are limited, use something else as the break out pattern. Otherwise, signal out of band in some other way (exception on end, have your caller test for "has next"), or use the "unique end ref" trick), with the usual implications of each of those.

[Limbic~Region]

nferraz,
Now, since I was studying how to create iterators using closures...

You might want to take a look at How To: Make An Iterator which I expanded a bit further for a perl.com article here. If you want to take it even further, you should take a look at Higher Order Perl by Dominus you are shown how to treat the iterator as a list for the purposes of filtering and transforming.

It is not too uncommon to have an iterator to an infinite list - what then should be the behavior if you ask for the entire list? I think a more sane approach would be to provide a parameter to ask for a specific number of fetches. Remember that $next->() doesn't always have to have an empty argument list. You could even have an optional start position so that you can fetch the next 10 values or the first 10. The amount of extensibility is really just restricted by the size of the memory footprint you are willing to let the iterator have.

[rinceWind]

Good call, nferraz

You are right, in my opinion, to leave the question open as to whether the idiom is useful or not. I agree with gaal that such an interface needs to be clearly documented.

However, I can see problems with using such an interface, as it is inherently ambiguous to the unwary. Can you be 100% sure that you are using it in a particular context? The perl builtins have enough traps for the unwary, see Is this a bug, or expected behavior? for an example. In terms of a popular module using your idiom already, the Class::DBI methods search and retrieve_all are an example.

What's wrong with having two separate functions, one which gives you an iterator and one which returns the list of values? Alternatively, if you are using an OO interface, return an object from ->new() which is basically an iterator with a $obj->next method, but also provide a $obj->all method to give the whole list.

I use this latter approach in File::Wildcard. To my mind, it's easier to document, to write test cases for, and does not place the burden of understanding the ambiguity with someone trying to use the module.

[gaal]

Not a bad technique, though one that needs clear documentation. FWIW, that's what glob does, too (except the iterator from glob is reset after it hits the end).

[dpuu]

You may be interested in Damian's talk, "Sufficiently Advanced Technologies", podcast at http://yapc.g.hatena.ne.jp/jkondo/. He mentions some refinements on this technique, such as returning an object with boolean overloads so that while ($iter) {...} will correctly set $_ on each iteration.

[salva]

there was some discussion on p5p some time ago, about the return value from subroutines like the one you are using:

sub foo { shift @foo if @foo }

[salva]

an iterator provides much less functionality that an array, for instance, you can only access the elements sequentially and you can not get the number of elements in advance. So, usually, returning an array is better than an iterator.

Though, there are some cases where an iterator is preferred: when you don't want to let full access to the array or when you are using another representation internally to hold the values like a db table and don't want to make a Perl array holding all the values for (memory) performance reasons.

For instance, In my module Sort::Key::Merger that merges sorted lists, I used the iterator aproach so that the original lists and the result list were not required to fit in memory.

[Jenda]

IMHO, if you already have the whole list then there's no point in returning an iterator, arrays are easier to handle and allow more operations. OTOH, if you checked near the beginning of the function and then either produced all the items and returned them in a list/array-ref or created an iterator that'll produce the items one at a time then it might be usefull.

Quite often though I believe you'll find the iterator too restricted sooner or later so you might use an object already. Even if for the start there's just a single method GetNext().

[aufflick]

It would be more obvious if Perl arrays were objects (as in Ruby), but remember that the whole point behind the iterater pattern is to be able to do what Perl arrays give you for free.

What do you get by returning an iterator that you don't get by returning an array or an arrayref?

The only thing I can think of is that you can call a "next" method that is non-destructive (unlike shift which is destructive to the array). If you return an array you could just return a copy if you don't want the recipient to destroy your array, but that would be ineficient for large arrays.