RFC Magic::Attach

[shotgunefx]

I've created a module Magic::Attach. It allows you to attach one referant to another.

The purpose of which is to allow a sub or any other type of object to attach some private data to a reference. This could be for lazy evaluation of arguments or whatever purpose you can think of.

When you attach a ref to another ref, you can optionally specify a nameid to avoid collisions. Multiple attachments are fine but other modules that implemented their own magic using the '~' user extension type may clobber us if they are less careful.

The module can be found here

This is my first real forray into XS so I'm sure there are things that could be done better. Comments and criticisms welcome.

It works with 5.6.1 and above (possibly with earlier versions but I haven't confirmed this.) The naming of the routines may change. (Would bind/bound/unbind be better suited?)

My other thought is if this is a generally useful module, perhaps the "optional nameid" shouldn't be optional (Currently it defaults to Magic::Attach). I worry that if this finds use and the default nameid are used in two different places it will lead to very, very subtle bugs. I can't speak for anyone else but before embarking on this, I would never think to look at "magic vtables" for problems.

I'd really appreciate any feedback on the implementation, usefulness, etc of this.

$attached = attach_var($target_ref, $attach, [$optional_name]);
$attached = attached($target_ref, [ $optional_name ]);
$attached = unattach_var($target_ref, [ $optional_name ]);


# A few trivial examples. 
sub elements(\@$){
    my ($ref,$size) = @_;
    $size = 1 unless $size;
    my $info = attached($ref,'elements') || attach_var($ref,[0,$size],
+'elements');

    my @ret = ();

    if( $info->[0] >= @$ref ) {
      unattach_var($ref,'elements');
      return;
    }else{
      @ret = @{$ref}[$info->[0]..($info->[0]+$info->[1] -1)];
      $info->[0]+= $info->[1];
    }
    @ret;
}

while( my @els = elements(@large_array,4) ){  
    # Process non destructively
}

# Lazily generate a span of integers.
sub int_span($$){
    my $info = attached(\$_[0],'INT_SPAN') || attach_var(\$_[0],[$_[0]
+,$_[1] ],'INT_SPAN');
    unattach_var(\$_[0],'INT_SPAN') and return  if $info->[0] >  $info
+->[1];
    return $info->[0]++;
}

while(my $i = int_span(1,1000000000)){
  print $i,"\n";
}

Thanks,
-Lee

"To be civilized is to deny one's nature."

---

update
The "magic" refered to is internal to the perl interpreter. See "Magic Variables" in perlguts for more information.

---

update Jan 21 one more example.

# Persistant subroutine data
sub somesub {
    my $p_args = attached(\&somesub,'somesub') || attach_var(\&somesub
+,{called=>0},'somesub'); 
    ++$p_args->{called};
    # The persistant arg hash could be keyed on the caller info if des
+ired
    # for context sensitivity
}

[theorbtwo]

DOCUMENTATION!

After reading though the examples several times, I think I understand more or less what this does. I shouldn't have to do that.

I vaugely know what magic is, and what the ~ magic vtable is. I had to read over the samples three times before I understood what you're doing there.

It doesn't help that the ~ magic vtable has nothing to do with the "traditional" meaning of magic -- that is, it's never automaticly called.

I wonder how difficult it would be to allow you to set other types of magic to a perl sub, to be able to create partialy-tied things, and other wierd stuff.

Oh, and I think SvREFCNT can be gotten from several other places as well.

---

Warning: Unless otherwise stated, code is untested. Do not use without understanding. Code is posted in the hopes it is useful, but without warranty. All copyrights are relinquished into the public domain unless otherwise stated. I am not an angel. I am capable of error, and err on a fairly regular basis. If I made a mistake, please let me know (such as by replying to this node).

[shotgunefx]

Point taken. I was intending to document it better after the feedback from this post (I assume I'll be making much changes)

"It doesn't help that the ~ magic vtable has nothing to do with the "traditional" meaning of magic -- that is, it's never automaticly called."

Actually if callbacks are setup they are automatically called. The only callback I am installing is for the freeing of the SV so that it won't leak the attached reference.

I do plan on another module to allow user-defined callbacks but I didn't want to get over my head. My grasp of things internal is still less than *COUGH* um, firm.

I added SvREFCNT for some additional tests I am going to implement. I didn't want to require any prerequisites for the tests alone.

Thanks,
-Lee

"To be civilized is to deny one's nature."

[theorbtwo]

Cool. All sounds good, though I wouldn't call it Magic::Attach, then... try Magic::ArbitraryData, perhaps? Magic::Attach::Anything? Magic::Tilde? Anyway, I'd save Magic::Attach for the other module, the one that attaches more traditional types of magic to a var.

---

Warning: Unless otherwise stated, code is untested. Do not use without understanding. Code is posted in the hopes it is useful, but without warranty. All copyrights are relinquished into the public domain unless otherwise stated. I am not an angel. I am capable of error, and err on a fairly regular basis. If I made a mistake, please let me know (such as by replying to this node).

[pdcawley]

Nice work. However, the "other modules might clobber you if they're not careful" part is a worry to me. When we implemented Pixie::Info as a way of storing out of band data for Pixie (we need it so that Pixie can manage objects in memory without having to alter anything 'visible' -- no invading of hashes for us...) we came to the conclusion that the very clobberability of '~' magic (and some of the special case semantics that are automatically associated with it mean that *not* clobbering '~' magic is hard work) meant we should use a different magic symbol, after all, there's over 200 available, we chose chr(155).

What we *should* have done, of course, is to hang a hash off the object and hang our out of band information in the 'Pixie' slot...

[shotgunefx]

Yes it worries me too. sv_unmagic() on '~' removes all '~' magic entries currently (Which is why the current implemenation does not call it). I wonder if there would be any way or (where) to determine what modules exist that use internal magic. I would think there are not too many. The only one I know of is Devel::WeakRef and Pixie.

I don't mind taking my own magic entry but worry about clashes with future versions of Perl 5.* and of course other modules developers.

-Lee

"To be civilized is to deny one's nature."

[pdcawley]

Just announce that you're grabbing a magic sigil on p5p, explaining why (that's what I did). Who knows, make it general enough and it might end up in the core.

[shotgunefx]

For the moment I am going to stick with '~' though it issues a warning if it finds any other '~' magic. (I'm now prefixing names)

Data::Attach Found a '~' magic that is not ours (EVIL ATTACH). Possibl
+e conflict with (Dat:attachtest)



-Lee

"To be civilized is to deny one's nature."

[John M. Dlugosz]

After the earlier brief discussion, I thought about the similarity between this feature and "properties" in Perl 6. I think it would be useful to design the front-end in such a way that it can be re-coded to use properties (mimicing the property syntax of Perl 6 directly is probably not practical without a source filter, and it may change before it sees first light anyway so it doesn't seem worth the effort, unless you want to specifically explore the Perl 6 feature in applied use before it's too late to redesign it).

I don't think the example at the end of your post is a good example. Why must your tuple be crammed into a simple scalar? That looks like the job for an iterator class based on a blessed array of two items, and possibly an overloaded stringize or numify operator.

—John

[shotgunefx]

I'm not up to speed on Perl 6. Between life and work I've been really squeezed. That's why I cursed you for asking that question :), if time permits I'll look into it. I do want to add callbacks though. One of my side projects is a perl-like language written in Perl. It's the basis for Yet Another Template Language. (Yes, I know it's been done to death) I want to avoid direct evaluation entirely. One thing that occured to me is that I could add a hook to any evaluated symbol or variable and use it to invalidate cached expressions when they are updated. (Most likely > half of the expressions won't change under iteration.)

I agree about the example, that's why I labeled them trivial. After the time I spent on this I was too fried and pressed for time to come up with something compelling.

Why must your tuple be crammed into a simple scalar?
I'm not sure what you mean by this. Do you mean that the positions are stored in an array ref? While I use iterators, I don't like the syntax. Most if not all of my motivation of getting into the guts of Perl is to try and bend it to my will :)

I've actually spent a lot of time trying to fathom the guts of Perl and it all started because of an iterator op I wanted to write and couldn't (The way it should work anyway) findone { coderef } @array.

Though the examples are not great, it is common to want to attach some data to another piece of data. We do this all the time with lexical hashes and stringified refs, etc. After your question, it just seemed like a cool thing to do that might be practical. I personally dislike ties, or anything else that makes me change they way I would write something or requires prep work on my end. I also don't want subs messing with their callers. So it just seemed like a good way to accomplish it while improving my understanding of internals.

Any thoughts on requiring a "nameid"? The more I think about it, the more I think it should be required. I didn't want to call it a "packagename" as that was confusing. nameid is pretty bad too though. Perhaps "slotname"?

-Lee

"To be civilized is to deny one's nature."

[John M. Dlugosz]

Yes, the property name should be required.

In exegesis 2, it shows that the new keyword is will assign a named property to a value: $val is Found(0) is sugary syntax for something like attach (\&val, Found => 0). For looking it up, it shows treating the plain scalar value like a hash. There's got to be a longer way, or it would have problems when the value is a hash. The Apocalypse probably shows more detail, but that's already changed so don't get too, er, attached to it. I think last I heard run-time properties used the but keyword.

But, Apocalypse 2 (end of the properties section) gives "a list of property ideas" that you might try using with your stuff.

—John

[PodMaster]

I see someone called you on the pod, good call ;)

Anyway, I do not feel the namespace you have chosen to introduce (Magic::Attach) is appropriate.

The Devel:: or the Perl:: namespaces are more appropriate.

Since the target audience of this module are perl developer's (that's how I see it), I'm leaning towards the Devel:: namespace.

I hope that helps.

Magic? ;]

MJD says you can't just make shit up and expect the computer to know what you mean, retardo! ** The Third rule of perl club is a statement of fact: pod is sexy.

[shotgunefx]

Thanks,
I'm open to suggestions on the namespace but if in practice, this proves workable and reliable, I wouldn't want it in the Devel namespace. I was thinking of it more as a general purpose tool. I know not everything in the Devel namespace is under construction of for introspection, but it does have that "beta" feel. As far as the Perl namespace, I've not run into any modules there. I'll take a look and see if that's more appropriate. Before it makes it to CPAN I'll be asking the modules mailing list as well.

-Lee

"To be civilized is to deny one's nature."

[shotgunefx]

I wonder if this couldn't be used for implementing co-routines. I started thinking about it and remembered a post by Ovid on the subject.

Here's what I am thinking and I am sure there are flaws and other issues problems I haven't thought of. You use attach_var() to create persistant arguments for the sub. (This could be wrapped in a routine, say persist(); This would just be syntactical sugar for $var = attached(blahblah} or attach_var(blabla);

You have a sub called yield that takes a block as an argument. It computes a "context key" from the caller info. It checks to see if it the calling sub has a "context stash" associated with it. This would be an array of contexts. If not it creates one (attached to the calling sub) and pushes an empty context.

It could then check to see if had been called already in this context. (With the same args, from the same line in source.)

If it has it returns. Basically a NOOP.

If it hasn't, it marks itself as being called in the stash and pushes a new context on the "context stack", it then executes the block.

You then have a sub called yield_return(). This pops the context off of the stack and returns.

It would look a little something like this. (Using TheDamians example at http://www.yetanother.org/damian/Perl5+i/coroutines.html)

package Tree;

sub next_inorder{
                my $self = persist();
                yield { $self->{left}->next_inorder } if $self->{left}
+;
                yield { $self };
                yield { $self->{right}->next_inorder if } $self->{righ
+t};
                yield_return undef;
}


# and later...
while (my $node = $root->next_inorder()) {
         print $node->{data};
}

Anyone see a flaw or potential snags with this attack?

-Lee

"To be civilized is to deny one's nature."

[adrianh]

Anyone see a flaw or potential snags with this attack?

Co-routines that call co-routines are where it gets tricky for the algorithm you outlined. Consider this line in your example:

yield { $self->{left}->next_inorder } if $self->{left};

next_inorder is called recursively, and the context of yielding a yielded result should leave the execution within the recursive call to next_inorder.

You could make this example work by differentiating between a yielded result and a normal return. However, once you have code that has side effects it starts getting hard.

[adrianh]

Comments/suggestions:

• I would name the subroutines attach and unattach rather than attach_var and unattach_var (they're attaching the reference not the variable.)
• Maybe have the name default to the current package, rather than Magic::Attach? I'd lean towards making it compulsary.
• This kind of thing:

  my $foo = attached($ref,'elements') || attach_var($ref,$something,'ele
  +ments')
  

  is likely to be common enough to encapsulate. Maybe:

  my $foo = attach_default($ref, $something, 'elements');
  

• Data::Attach rather than Magic::Attach? I don't think Magic is an appropriate top-level namespace... Devel::Attach would be good too.

Can't really comment on the implementation - I don't know the internals well enough.

[pdcawley]

Have we considered how nicely this would play with scalar attributes? One could imagine appropriate Attribute::Handlers hanging their data off arbitrary scalars. Then one could imagine:

sub UNIVERSAL::property {
   require Magic::Attach;
   my $self = shift;
   my $property_name = shift;
   return Magic::Attached::attached($self, $property_name);
}

sub UNIVERSAL::but {
   require Magic::Attach;
   my $self = shift;
   my($property_name, $value) =@_;
   Magic::Attach::attach($self, $property_name, $value);
}

and the like.

The fun begins when you start to try and implement Perl6ish tricks along the lines of return $value->but('true') or (possibly nicer) return $value :but('true'); . For this to work really well you would need have 'but' dispatch to appropriate handlers in some way (possibly through a registry system), but it's not beyond the wit of man...

Can you tell I've been thinking along these lines for a while but have lacked the appropriate tuits?

[IlyaM]

I think detach sounds better then unattach.

--
Ilya Martynov, [e-mail]
CTO IPonWEB (UK) Ltd
Quality Perl Programming and Unix Support UK managed @ offshore prices - http://www.iponweb.net
Personal website - http://martynov.org

[shotgunefx]

It was actually "detach" at first though I changed it because attach/unattach had a better ying/yang to me than attach/detach. Suppose I could alias it.

-Lee

"To be civilized is to deny one's nature."

[shotgunefx]

Just an update for anyone who is interested. I'm working on adding the perl side interface to the magic functions (get,set,clear,len,free). It almost works but I'm running into some random seg faults and core dumps. I hope to have the issues resolved soon. Possibly by this weekend.

The code needed to be modified to walk through the magic of a variable and dice up the linked list to remove instances of unattached magic as sv_unmagic removes all instances of a given type of magic. I believe this is where my problem lies. I suppose I don't actually need to clean up these structures as they do go away when the variable is released but it feels like a leak to me so I'm going to continue to try and plug it.

Hopefully there will be a working version posted by the end of this week.

-Lee

"To be civilized is to deny one's nature."