How A Function Becomes Higher Order" reflections on Perl 6

[John M. Dlugosz]

I just read How A Function Becomes Higher Order by Limbic~Region, as it was linked to from a current thread. When I got to the max function, I thought, "that's built in to Perl 6". So is a lot of that. Furthermore, it's instructive to see how Perl 6 adds the kinds of features that we would implement in Perl 5. It's not just a random collection of cool new stuff, but designed to learn from existing use cases.

First of all, max is a built-in binary operator. You can write $a max $b max $c. But you can also use any operator as a reduction operator. The original

use List::Util 'max'; 
my @scores = <FH>; 
my $high_score = max(@scores);

would become

my @scores = $FH;  # $FH.lines called automatically in list context
my $high_score= [max] @scores;

The original Perl 5 commentary continues, "Unfortunately, this requires all of the scores to be held in memory at one time and our file is really big." Well, in Perl 6, that is not the case! The $FH.lines function returns a lazy list, and doesn't have to suck in the whole file.

This is more profound than just making something a built-in when it was in a module before, and making the syntax cooler. This eliminates the entire reason for not using the simple function in the first place! This is equivalent in execution to the second listing, not the first: the while loop that reads a line at a time.

Aside: this is not addressed in the Synopses, but there is a difference between a lazy list and a forward iterator. We specifically do not want to remember all the elements in @scores as they are reified, as that would defeat the purpose of not reading them in. Hopefully, the compiler will see that @scores is never accessed again and will take care of that. But writing my $high_score= [max] $FH.lines; without the intermediate variable will assure this optimization takes place. Or, get the Iterator rather than the lazy array of file contents in the first place: my $it - $FH.Iterator;.

But now comes the hard part. How can we turn the lazy list into a "push" function, that is given a little data at a time? The use of feeds comes to mind. The syntax

my @feed <== 1, 2, 3, blah blah blah;

will create the lazy list that is something of a pipeline to the right-hand-side that generates the data. This causes multiple threads to be used, implicitly. My first thought was to wish for a feed that is "still open" so the consumer doesn't quit when empty, but waits for me to put more stuff into the pipeline, and I'll explicitly close it when I'm really done. Something like this:

# bad - do not use
my Feed $pipeline .= new (... options to make it do what I said above 
+...);
my $result= &[max] <== @$pipeline;     # ??? threaded on both sides ?
# blah blah blah
         $pipeline.add ($item);   # every once in a while

But casting around for the correct syntax and meanings at every turn, I realize this is not the way they are meant to be used. But the second half of the above makes me think of gather/take.

my @pipeline <== gather {
   while $whatever
    # code here does its stuff and eventually issues:
       take $item;
       }
   }
my $result = [max] @pipeline;

The <== puts the RHS into a thread that it can execute in the background or resume when it needs more elements. The array is a lazy list that reifies from that background routine when accessed. Again, this is as described in the synopses but what I really want is an Iterator, forgetting the first part of the list after it is used.

The gather/take is a general way of collecting results that are not simply the result from a loop, but may be produced at some point within the loop, or at several points, or whatever. The stuff you "take" is put into a list that is returned by the gather statement.

This is still the other way around from the original. The "push" becomes a generator, so the "pull" usage still works. With multiple threads in a producer/consumer relationship, there is no push or pull, but simply cooperation. Since the generator code is a closure in the same context with the code setting it up and using it, there should be no problem writing it this way. The "threads" are more interesting than threads you are used to, as it will behave more like a co-routine with access to the calling stack just like a regular function call, not like a pinched-off bud on a new context stack.

Meanwhile...
Another problem illustrated by the code is "named parameters".

sub gen_reduce {
    my $usage = 'Usage: gen_reduce("initial" => $val, "compare" => $co
+de_ref)';

    # Hashes need even number of arguments
    die $usage if @_ % 2;
    my %opt = @_;

    # Verify that compare defined and a code reference
    die $usage if ! defined $opt{compare} || ref $opt{compare} ne 'COD
+E';
    my $compare = $opt{compare};
    my $val = $opt{initial};
    # FINALLY start the actual function

We all know that Perl 5's parameter passing is too rudimentary. The first Apocalypse promised good function calling, and that was subsequently filled out beyond anyone's wildest dreams. I could force the use of named parameters in the call, but I'll make them positional for reasons to be shown next. The caller can still call them with names and not worry about the order.

sub gen_reduce (&compare, ?$initial)

That's it. The compiler checks for one or two arguments (the ? means optional) and that compare is a Callable object. You don't have to. It can even spot mis-uses at compile time, like in other languages.

So the caller may still write: gen_reduce(compare->$comp), but because the function takes a parameter declared using the & sigil, you can use it more like built-in statements:

my $maxstr = gen_reduce { 
   return length($_[0]) > length($_[1]); 
   }

But the parameters within that function are pretty ugly still. Perl 6 can do better:

my $maxstr = gen_reduce { 
   return length($^left) > length($^right); 
   }