Is Devel::Declare really so much less evil than source filters?

[moritz]

Recently there has been quite a hype about syntax extensions to Perl that don't work via traditional source filters, but through Devel::Declare.

Lots of people who wouldn't touch source filters now write such syntax extensions like MooseX::Declare, and they seem to be well accepted.

However I still have some measure of reserve towards them, which I'd like to discuss with you. There are basically two reasons why source filters are wildly considered very evil™:

1. Source filters have to parse Perl code. This is very hard, so sooner or later they fail, leading to errors which are very hard to track
2. Since source filters have to parse Perl code themselves, you can't generally combine two source filters, even if they change different parts of the syntax

Now I read that Devel::Declare only hooks on barewords, and use the built-in parser for the rest.

But still the syntax changing module has to parse something. For example MooseX:::Method::Signature allows things like:

method hello (Str :$who, Int :$age where { $_ > 0 }) { 
  ...
}

It is my hope that the where-block allows arbitrary Perl code. Is that true? And if so, is the "real" Perl parser used for parsing it? And if yes, can other syntax changes based on Devel::Declare be used in this block?

If the answer to all three questions above is "yes", then I'm happy to acknowledge that Devel::Declare is really a good thing, and worth using (and actually comes quite close to "real" macros, except in scope of what can be changed).

If not, I'd argue that D::D based syntax changes are only gradually less evil than source filters, not conceptionally less evil.

What do you think? Does anybody have experience if D::D based syntax changes are really robust?

Update: tye++ pointed me to the reviews of TryCatch, which says

Great module for try/catch sugar in perl. I love pairing it with Moose::Util::TypeConstraints to have type based exception handling. Make sure that you have the latest Devel::Declare or you can have some weird issues that won't seem to make any sense.

The last sentence seems to suggest that it's not robust, or at least it was not. Still the question remains if it's a conceptional fault, or merely a lacking implementation right now.

[Corion]

From my reading of Devel::Declare, it uses (and exposes) the same parsing routines that Perl uses, so for subexpressions, it doesn't have to reimplement that. In that way it sure is better than Filter::Util, which has to reimplemented Perl string parsing and POD parsing.

As the code generation is (conveniently) still string-based, I imagine fun things could happen if you inject string snippets that need parsing by another Devel::Declare-based filter, as you're then dependent on these filters having been loaded in the right order. But maybe Devel::Declare invokes the filter chain in a loop so you can send of Perl into an infinite loop by having mutually recursive filters instead.

[stvn]

Is Devel::Declare really so much less evil than source filters?

The short answer is "not really, they are still quite evil".

Source filters are fragile because they have to parse Perl (as you said), Devel::Declare sidesteps this by using Perl's own parser. But this is just one of the reasons that source filters are fragile. Extracting the new syntax from the Perl code is just one step in the process, you then have to parse it into a usable form and generate code from it. These steps are still present in Devel::Declare and are by no means trivial or simple steps, especially given that your generated code must then be injected back into the parse stream of an existing document.

Now, that said, if Perl had a good set of Parser/AST/Code-gen libraries, then I think my response would be "yes, they are *much* less evil".

As for the question of "Too Much Magic", well, this is true, but being made less and less true as the Devel::Declare team petitions p5p to get some of their hooks made official APIs. How far that will go, we will have to wait and see.

Great module for try/catch sugar in perl. I love pairing it with Moose::Util::TypeConstraints to have type based exception handling. Make sure that you have the latest Devel::Declare or you can have some weird issues that won't seem to make any sense.

The last sentence seems to suggest that it's not robust, or at least it was not. Still the question remains if it's a conceptional fault, or merely a lacking implementation right now.

I wouldn't put too much stock into CPAN ratings, they are sometimes really insightful and useful, and other times are the rantings of lunatics with little or no understanding of the issue. In this case I suspect that the comment speaks less to the overall fragility of Devel::Declare, and more to the fact that Devel::Declare is still a new module and still needs to have all its bugs worked out.

[tye]

After more than a day, it appears that the blog author might never approve my reply to his reply. So I'll just post it here:

> > It is the bane and hallmark of ETOOMUCHMAGIC

> Actually, all of this is the bane and hallmark of EXPERIMENTAL. We're a small group of people.

Ordinary bugs are the bane and hallmark of new work.

"some weird issues that won't seem to make any sense" is the bane and hallmark of ETOOMUCHMAGIC.

The great difficulty in giving good errors in the case of failure is indeed another reason why I avoid complex magic.

And I've posted it to this part of the thread because stvn said:

In this case I suspect that the comment speaks less to the overall fragility of Devel::Declare, and more to the fact that Devel::Declare is still a new module and still needs to have all its bugs worked out.

I wasn't complaining because the module (horror!) had a bug at some point. I was noting that the description of the bug was quite telling. It sounded exactly like the complaints that come in when people are using Switch.pm. I don't hear things described that way other than when too much magic is being invoked. And I don't believe the blog author really refuted that characterization.

The blog author admitted that when things go wrong (people will make mistakes, even when using Devel::Declare) the resulting error message may be not "decent". The (likely more honest) CPAN review said "weird issues that won't seem to make any sense". Those both smell very much like ETOOMUCHMAGIC.

And I wasn't really even talking about "fragility". ETOOMUCHMAGIC doesn't always make things "fragile" as in "easily or frequently broken", though that can certainly be one of the unfortunate consequences. The more characteristic problem with ETOOMUCHMAGIC is that when it does break something, the breakage can be quite cryptic and fundamental and might not even hint at the module that is pumping in the most "magic". So you waste a lot of time trying to find the source(s) of the problem. Understanding the failure very often is extremely complex. And fixing the problem very often ends up being very impractical (because the breakage is fundamental to some deep aspect of the complex magic that is hiding behind the "subtle" exposed interface).

About a year ago, I was mentoring somebody who came to me with a very strange Perl failure. They were implementing some testing tools that actually made use of quite a few different pieces of magic. I was pretty firm in pushing them away from some forms of magic that I considered too problematic or just very much not worth the benefit (and I found I had to keep repeating these pushes because this developer really did want to solve problems thoroughly).

Anyway, the failure provided very few clues as to a likely culprit. I told him to replace the XS-based JSON library with a Perl-only equivalent. He pointed out that the failure was happening in parts of the code that weren't using JSON and weren't even "close to" the parts that were using JSON. I told him to humor me. The problem went away when he replaced the JSON module.

I was quite busy with other tasks at the time and so did no diving into the problem and would not have looked forward to such a dive. It looked to me like such a dive would have taken a lot of time and concentration and knowledge and then might not have been successful in actually identifying a problem much less finding a fix for it.

My instinct of "remove the most magic piece" served me well that time (it often does). Luckily, this particular project was young and it was easy to rip out that piece. Relying on subtle magic is a recipe for pain, IME.

By the way, as I noted, this testing tool required quite a few pieces of magic. So many pieces that I would never consider it an acceptable tool in a Production environment. I pushed hard to completely avoid the heavier magics being considered and to make even the moderately interesting magics optional (and part of the motivation for creating this tool was that the existing "state of the art" tool was unconditional in its use of some fairly mild-sounding magic that often isn't needed and that people had complained about having problems with). I think the tool will be invaluable in a testing environment (I changed jobs after that and haven't yet had a chance to put it to good use). And I'm also quite sure that some of the magics used will cause problems in some situations.

I use magic. But I'm pretty strict about avoiding building things on top of magic. It is experience that led me to such caution. Now I always look for the warning signs of magic.

[stvn]

Let me start out by saying that I personally do not use Devel::Declare or any of the modules which use it (MooseX::Declare, TryCatch, etc). My reasons for not using it are pretty much the same as your reasons. To be specific, I think that MooseX::Declare pukes up too many internal details when it errors and this makes for very tricky debugging and fixing that seems to be a non-trivial task (I say this only because no one seems to have fixed it yet or even come close).

This too is exactly why MooseX::Declare remains a MooseX:: module and is in no way an "official" part of Moose. Not until I am sufficiently convinced of the stability of Devel::Declare and the error messages of MooseX::Declare greatly improve will it even be considered.

In short, I totally agree with you (horror!), magic should be used sparingly and is never a foundation upon which you should build.

But, all that said, the Devel::Declare team is trying to reduce the magic-ness by working with the p5p folks. What will come of this? I have no idea, we will have to wait and see.

[BrowserUk]

So far I'm less than impressed.

• The docs are awful. Mixing up builtin apis with documentation-only 'add-ons'.
• The interface is eclectic to say the least. Some apis have to be called as class methods, others as subroutines. Neither are documented.

  setup_for() & toke_scan_word() have to be called as class methods, all the rest (that i've tried) as subroutines.

• It kinda works if all your syntax is contained within one line, but gets messy (if not impossible), if it spreads over multiple lines.
• It doesn't handle nested syntax. Eg. switch( $n ){ case 1:{...} case 2:{...} }

  It has a routine to parse nested tokens, but if you replace the switch leaving the cases as is, it doesn't call the case handler.

  If the syntax above is spread over multiple lines, you can replace the header of the switch okay. Then the case handler is called, but there is no way to tidy up the closing brace of the switch.

Source filters are easier to write and understand. Shame. It looked very interesting (to me).

[LanX]

Hi

this thread initiated an interesting discussion about macros, parsing and sideeffects in the chatter box which I would like to document for the afterworld:

please send me a /msg if you want to be excluded from the log

Read more... (6 kB)

UPDATE: reversed text, still working on better formatted version.

[tye]

I expanded my remarks in a comment to http://www.catalyzed.org/2009/05/dawn-of-a-new-age-in-perl-how-develdeclare-extends-perls-syntax.html.

[LanX]

for those interested here is the script I used to transform the CB-log saved from http://pthbb.org/cb/last.cgi into a readable post.

open $cb,"<","/tmp/cb.txt";
 
#- whole conversation into one string
my $slurp=do {local $/,<$cb>};<P>

#- split into single chats
my @chatters=split /^\s*$/m, $slurp;
 
for my $chat (reverse @chatters) {
 
  my (undef,$head,@text)=split /\n/m, $chat;
 
  #- print head
  my ($date,$time,$author)=split /\s+/,$head;
  print "<tt>", "[[$author]] $time", "</tt><br>\n";
 
  #- /me text to italic
  if ($text[0]=~s/^\/me/[$author]/){
    unshift @text,'<i>';
    push @text,'</i>';
  }
 
  #- named links
  for (@text){
    s# (\w+) \( (http://.*?) \) #[$2|$1]#xg
  }
 
  #- print text
  print join "\n",@text;
 
  #- print separator
  print "\n<p>\n\n";
 
}

[BrowserUk]

Update: Answered my own question. It has to be Devel::Declare->setup_for(...) not Devel::Declare::setup_for(...)

Does anyone know what do I have to change in this:

package Cswitch;
use strict;
use warnings;
use Devel::Declare;

sub import {
    print "Import called with @_";
    my $class = shift;
    my $caller = caller;

    Devel::Declare::setup_for(
        $caller,
        {
            switch => { const => \&pSwitch },
            case   => { const => \&pcase },
        }
    );
    no strict;
    *{ $caller . '::switch' } = sub () { };
    *{ $caller . '::case'   } = sub () { };
}

sub pSwitch {
    print "pSwitch: @_";
}
sub pCase {
    print "pCase: @_";
}

1;

In order that this would cause the parser subs to be called:

#! perl -slw
use strict;
use Cswitch;

switch 1, {
    case 1, {
        print 'Got 1';
    };
    case 2, {
        print 'Got 2'
    },
};

to stop this?

C:\test>t-DevDec.pl
Import called with Cswitch
Number found where operator expected at C:\test\t-DevDec.pl line 5, ne
+ar "switch 1"
        (Do you need to predeclare switch?)
Number found where operator expected at C:\test\t-DevDec.pl line 6, ne
+ar "case 1"
        (Do you need to predeclare case?)
Number found where operator expected at C:\test\t-DevDec.pl line 9, ne
+ar "case 2"
        (Do you need to predeclare case?)
syntax error at C:\test\t-DevDec.pl line 5, near "switch 1"
Execution of C:\test\t-DevDec.pl aborted due to compilation errors.