Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Free-Form Delegated Behavior

by John M. Dlugosz (Monsignor)
on Apr 22, 2011 at 04:21 UTC ( #900763=perlmeditation: print w/replies, xml ) Need Help??

I was looking at the Marpa module and one thing about the API made me think "this needs to be improved". However, I'd like to reflect on the general class of problem involved, and discuss the best way it might be handled. Or, expose any limitations or tradeoffs in such methods. This might serve as a model for those who run into the same problem later, and hopefully will inspire the author (Jeffrey_Kegler) who is still working on the final form of the module's API.

Says Jeffrey:

I don't like inheritance, but we can get into that in the exchange. I'm likely to learn something, and others may profit as well.
The problem, in its general form, is that of supplying possibly large amounts of "callback" information, and furthermore the exact callbacks needed are not pre-determined. So I call it "Free-Form Delegation".

If you had one function that could be supplied by the caller (or object's configurator), you would just use a function argument. If you had several that were rarely but possibly overridable from the default behavior, being able to override known functions in the class might be the simplest for the author and easy to do when needed. If the behaviors were not rarely altered but a necessary part of the object's use, a delegated object might be easier to use than a derived class, but we know in Perl they are largely equivalent. Anyway, we can discuss those and sling it out.

In strongly typed languages, you can define an "interface" that the supplied implementation will use to supply all the functions that will be called.

But what if the names of the functions to be called are not known, in general, to the object? In Perl this is not a game-changer in terms of how we supply that functionality, but it might still be a conceptual shift in just what it is we are doing. In Marpa, these are the "actions" called by the parser. Which actions even exist are also part of the configuration, and are not pre-determined by the object we are supplying it to. That's why I call it "free form": we don't know the functions or their relationship to each other in advance.

Now Marpa supports two related ways to do it, which I think follow the road from the simplest toward an OO solution. The simplest uses Perl primitives that seem to be good for this task: The name of a package is given to Marpa. Marpa looks for all the named actions in that package. To deal with interrelationships, a stash is passed along with the specified callback data, in the form of a simple hash reference.

The second form refines this idea a little: Instead of a plain hash to use as a stash, it will call new in that package and use the resulting object as the extra argument. Hmm, now the callbacks look like the way Perl does Objects, passing $self as the first parameter. Except, he's still forming the callback by explicitly looking for a named function inside that package, not using Perl's dispatch mechanism. That's what I think is "wrong" or at least missing a feature.

Now the action named might not be present. So we don't want to just code $object->$action(@args), as it will try other things if the named action is not a function. So, if memory serves, the right way to code this is with can. In the early days, it was widely known that the return value was the method resolved, but that was not guaranteed! What is the proper, kosher, way of looking up what a dispatch would do without calling the function just yet? (Edit: chromatic's book shows using can to keep the return value as a code reference. ) The proper way would, of course, work with plain Perl blessed references as well as Moose and other fancy stuff, and overridden dispatchers and everything. There really should be an approved primitive, right?

Let me digress a moment and explain why I want inheritance to work. Suppose I have a base grammar that I parse with Marpa, but several options that turn different productions on or off. In fact, my translator object (built using Marpa internally) might provide its own extension mechanism that causes it to generate new actions and productions based on the configuration it reads. In more general terms, whatever solution is built using the object that takes free-form delegation might itself be derived from (problem-wise, not necessarily literal class derivation). To extend or alter the callbacks without messing up the original, inheritence does the trick!

And, given the model described above, why not use can instead of direct hard-coded package namespace lookups? Since it almost matches the use of Perl objects already, and the relationship of the callbacks will treat their "stash" like methods on the same instance, it would surprise people that inheritance wouldn't work.

If all the callbacks are expected to be there (or inherited from a supplied default implementation!), just calling $object->$action(@args) would do the trick. That's certainly easier than poking around in symbolically named packages at run time. But Marpa illustrates some further complications.

It might look for several different names: the can form handles that just fine, looking for $name1 and if it's not present trying $name2 etc. But in that case, the final function chosen is indeed a method in the class.

Another wrinkle is that the code called might be "somewhere else", not a method of the class. A straightforward thing would be to specify a code ref, rather than a name, for the Action. Now with classic Perl 5 objects, calling $object->$action(@args) still work just fine if $action is a code reference. It doesn't do any lookup but just calls it with $object as the first parameter. With fancier object systems, this still works, right? I don't see any reason why the code in that ref can't turn around and call methods on its first parameter.

If the ref passed in is not declared as an anonymous sub or other sub that simply expects such an argument, it might be more awkward. For example, passing in a ref to a method of a different class, where that class is created using some fancy object system. Well, don't do that. You can always pass a closure that calls the method on the desired class.

So, such a free-form delegation system can support names of methods (dispatched in the normal way) and code ref pseudomethods. Anything else can be handled, easily, using that. Now comes a deeper question: is this even the right approach? We followed a path starting with the most obvious use of a primitive Perl 5 feature (packages as collections of named functions) and followed basically the same road that Perl 5 objects did, to passing an object that will use whatever kind of dispatch the object is defined to use. But is that the wrong road? Is there another totally different way to pass in a bunch of free-formed delegated behavior? I think the requirements are the same as that used to make an object system, so the answer is no, and furthermore reinforces the high-level solution: pass an object and don't worry about how the object works internally.

So let me conclude with a straw-man recommendation for how such an API should look. Given is that other configuration information includes action names. Include a setting, say $d, which may be a string or a reference. If it is a non-code reference it is used as-is, $obj=$d, for later calls to $obj->$action or the more advanced can probes. If $d is a string, then $obj= $d->new.

For simple cases, where you don't need state between calls, or just want to use a simple scratchpad, you could allow new to not exist. In that case, it pretends that you wrote a simple new that just blesses an empty hash reference. That way the dispatch on the resulting $obj still works and you don't need different code to deal with plain packages. (in the Alpha version, Marpa has different variables for specifying plain packages or classes to be constructed) Finally, if it is code ref, it is called to return $obj. This deals with the case where you want to pass custom arguments to new (just write an anon sub around the call), use a different constructor function, clone another object, or whatever.

Extending the system a bit, I address my only concern: that it may call functions not meant as Actions but are there for internal use or leftover from other code that was harnessed. When the action names are static and listed in the configuration along with the class to call, it is not something worth worrying about. But if the call-ability is carried forward to other code that "discovers" the available callbacks, we want to be more explicit about what is to be called! Consider a plug-in for a scriptable system, for example. You list the available commands and it tells you all kinds of stuff that you can call if you so desire.

In this case, I'm thinking an attribute on the sub would do the trick. That would also let you classify different entry points for different purposes. For example, sub dir :Command { ... and sub mungeit :Filter { ... .

So, any thoughts on any of this?

Or at least help convince Jeffrey to use something like my strawman proposal and not the brutal direct lookups which don't inherit?


Replies are listed 'Best First'.
Re: Free-Form Delegated Behavior
by pemungkah (Priest) on Apr 22, 2011 at 18:04 UTC
    Way back when I started on something similar to this called Class::AutoPlug. The idea was to be able to transform a class that didn't accept plugins to one that did with the minimum possible amount of code. It did use attributes to add methods, plus pre- and post-hooks for existing or added methods.

    It has a number of shortcomings, notably that the plugin installation order is not defined. This makes it difficult for one plugin to re-delegate another, since there's no guaranteed order. It would of course be possible to add this via more attributes, but I moved on to another project; this one languished and has now suffered bit rot - it's currently no longer passing its tests on 5.10 and 5.12. Take a look if you'd like; it's on github here - if it's useful, feel free to fork and play with it. I'd be happy to pull any changes or fixes back.

      Interesting. If I understand that correctly, the gist of it is explained with this: Causes the local_impl() subroutine to be added as the method method_name in the pluggable class. That is, it adds methods to a class.

      That sounds a lot like Roles in Perl 6 or Moose. So today I could just add a role to a class using with within the class (or after the fact if it's not been sealed), or even do it on a per-object basis. The latter doesn't have supplied syntactic sugar in Moose but is a primitive in Perl 6.

      Your implementation has an interesting feature in specifying which methods are injected into the host class, rather than just taking all of them. That means within that method, a dispatched call will see the host object, but a regular sub call will be compiled in the scope of the package it was actually declared in. It can call non-virtual helpers that don't become overridable in the host class.

      I think as a means of changing/extending the functionality of a class, this is the same as deriving a new class. Roles and plug-ins just allow such units of extensibility to be packaged and reused easier.

        Sorry to be so dilatory in getting back to you on this - yes, the concept is essentially to allow you to arbitrarily subclass in a particular way. It grew out of the hand-implemented version of this in WWW::Mechanize::Pluggable, which was designed to get around the problem of proliferating subclasses of Mech, or in having a single big heterogenous subclass to add all the features I wanted.
Re: Free-Form Delegated Behavior
by BrowserUk (Pope) on Apr 25, 2011 at 17:31 UTC

    I've read your post several times now and I keep coming back to the same question: How do you write code to use methods you don't know the names of?

    With normal inheritance, you write the code in terms of the base class methods, and the subclass overrides (or just provides) the implementation. But the method names and parameters are known up front.

    What you appear to be describing is a mechanism for constructing classes or objects at runtime and "discovering" the methods available on the fly. And your concern is how to safely invoke the methods so discovered.

    My concern would be how could you ever write code to use such an object?

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Look at the synopsis for Marpa. It shows a simple grammar that triggers do_add and do_multiply actions, and supplies all the actions in a package called MyActions.

      Now suppose I want to derive from this grammar. I add a production that calls do_exponent, which is easy to do: just push it on the list of productions, or call rules again on the existing parser instance to add another rule.

      Now how do I add the implementation of do_exponent? I don't want to add it to the MyActions package, although Perl supports injecting it without altering the original source file. That's the problem I ran into with the Alpha API.

      Now if the Action methods in MyActions were already acting like true methods, using the first argument as an instance and doing more stuff with it (say, to built a tree representation of the parse), it makes sense that my new do_exponent function would also want to call the (inherited!) helpers to manipulate and prepare the tree nodes.

      Knowing the difference between public API functions and private internal guts is a general Perl problem not unique to this. You see a class, you can write code to call anything willy-nilly, since it doesn't have protection. Just don't do that! Call the documented methods for their intended purpose. In this grammar example, the Actions named in the grammar are no different than normal code written to use a class. I code calls to methods I know about, and don't code calls to "other stuff" I found from looking through the source file or the package symbol table.

      My suggestions for the Marpa API in particular don't deal with "discovery", and is no different from any other module in that you write code that uses the methods you discover only by reading the documentation.

      Extending it to "discovery", like with Catalyst that lets you write methods that will be bound to URLs and called by surfing the web, is more general and not needed by Marpa in particular. But if there was a general module that "did the callback behavior", and it supported tagging the callable actions, that would be just fine.

      Hmm, I seem to have strengthened my argument that the Actions in Marpa should be done with an object. When using Marpa, you also code a module/class of some kind yourself, and code calls to that module. It's like any other module usage, only the code that calls the entry points doesn't look like typical Perl code but is highly declarative.

Re: Free-Form Delegated Behavior
by Jeffrey Kegler (Hermit) on Apr 26, 2011 at 06:14 UTC

    I apologize for taking so long to reply to John's thoughtful post re Marpa and its interface. My excuse is that, while researching the issue and mulling over what John had to say, my answer changed several times.

    I'm much more sympathetic to the idea of incorporating inheritance into Marpa's semantics than I was at first, for two reasons. First, grammars in Perl 6 allow inheritance. When it comes to interfaces I try to imitate Larry Wall, even when I'm not at all sure why he's doing what he's doing. I find if I don't imitate Larry, I often end up learning that he was right the hard way.

    Second, rereading John's post, I do see his point -- the interface really does seem to beg to be extended just that teensy extra step to methods and objects. And these days, once you've added methods and objects, the nose of the proverbial camel is under the flap of the tent and you may as well start figuring out where to put the rest of the animal.

    I have four reasons not to head down this road immediately:

    1.) It would definitely add complexity to the interface for the semantics, already the most complex part of Marpa's interface. For a new module especially, a dauntingly complex interface is a sure-fire way to delay acceptance.

    2.) Object-oriented is not a major research interest of mine. I am quite happy to let others take the lead in it. When it comes to inheritance, I am even less enthusiastic. I have no desire to impose these preferences on others, especially if doing so would delay the acceptance of Marpa, but it is my research interests which guide my priorities.

    3.) Marpa currently allows even anonymous subroutines in its semantics, via an override mechanism. In fact, anonymous closures, generated at run time, play a major role in at least one Marpa application. For this override mechanism, it is very handy to have a well-known and easily determined name for every function used in the semantics. What John (with a certain irony) terms the "brutal directness" of my naming mechanism is, in this context, a feature. This feature would need to be preserved or replaced with something just as good.

    4.) At this point, the internals of Marpa are likely to develop in a way that offers new opportunities for interface experimentation. Most of my work is on putting the core algorithm of Marpa into a highly optimized C library -- libmarpa. libmarpa is almost totally agnostic on interface issues. An obvious step beyond this would be to create a new, "thin" Perl XS interface to libmarpa, so that users can create their own Marpa-based parser generators. If in doing this I wound up out of the interface business, I would deem it an outcome that was devoutly to be wished.

    Here's a suggestion for someone interested in experimenting with a new semantics for Marpa: As John notes, all Marpa's semantic actions have a first argument which can be created by a package constructor, and this is already a kind of "self" argument. A package constructor in Perl can in fact return anything -- including an arbitrarily overloaded object blessed into a package which has nothing to do with the package that contains the constructor. It could be arranged that all semantic actions call another method on this object, passing on their arguments. All this comes at a modest cost in time and a higher one in elegance, but it would allow someone so inclined to experiment with arbitrary complex inheritance object-orientation schemes while using the Marpa parse engine.

      while researching the issue and mulling over what John had to say, my answer changed several times.
      That is a good "meditation" indeed!
Re: Free-Form Delegated Behavior
by Anonymous Monk on Apr 25, 2011 at 14:54 UTC
    I hate to encounter code which makes me say, "I am looking at this statement and I do not know what it does." Even worse when "I cannot know for certain what it does." Even even worse when "this statement might be executed a thousand times and do a different thing each time." Because I know that the problem that costs time and money is not, trying to figure out what the code is supposed to be doing, but, figuring out what the hell it just did and why.
      I don't understand how this relates to the post, other than to advocate "least surprise".

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://900763]
Approved by Corion
Front-paged by Corion
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (3)
As of 2018-05-22 06:43 GMT
Find Nodes?
    Voting Booth?