Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked

Perl 5 Optimizing Compiler, Part 9: & The Low-Magic Perl Commandments

by Will_the_Chill (Pilgrim)
on Nov 08, 2013 at 12:58 UTC ( [id://1061711]=perlmeditation: print w/replies, xml ) Need Help??

Howdy Monks,

I've been hard at work, tracking down all the different kinds of magic in Perl 5, and figuring out how I can disable said magic so that our RPerl compiler can make low-magic Perl 5 code run as fast as C. (Yes chromatic, AS FAST AS C.)

I'm proud to announce the launch of The Official Website, which now includes:

RPerl Frequently Asked Questions

RPerl Performance Benchmarks

The Book Of RPerl

The Low-Magic Perl Commandments

For your convenience, I've included a copy of The Low-Magic Perl Commandments below.

Brownie points go to any Monks who can find even the tiniest bits of Perl 5 magic missing from this list compiled by bulk88, rurban, mst, and yours truly!

HTAAOF Brothers,
~ Will the Chill AKA The Voice In The Wilderness

The Low-Magic Perl Commandments

(Hidden as spoiler to reduce post length.)
  • Comment on Perl 5 Optimizing Compiler, Part 9: & The Low-Magic Perl Commandments

Replies are listed 'Best First'.
Re: Perl 5 Optimizing Compiler, Part 9: & The Low-Magic Perl Commandments
by dave_the_m (Monsignor) on Nov 09, 2013 at 09:35 UTC
    Perl without its internal 'magic' implementation is essentially no longer Perl.

    Here's some of the features of perl that would no longer work in sections of code where magic was disabled:

    $1, $2, et al, and in fact most other special variables, such as  $., $!, %ENV, %SIG, @ISA etc;

    while (/.../g) (/g match in scalar context);

    weak references;

    ties; so for example this stops working: use Config; if ($Config{foo}) ...

    the taint mechanism;

    $#array, keys %hash, substr, vec in lvalue context;

    foo($hash{nonexist}), where the element is only vivified if foo() assigns a value to $_[0];

    It would also interfere with offset caching on utf8 strings, meaning that for example //g on long strings will go quadratic on length.

    And of course if you write a function which is allowed to be called by others, then you have to allow that @_ may contain magic values which won't work properly if you've disabled magic in your function. Similarly, the return value you receive from other functions or perl builtins and ops may contain magic.

    In short, all but the most trivial of perl code requires magic to be enabled.

    Note also that you can't have just 'low' amounts of magic; in a particular section of code you either perform a check on each SV before you access it to see if it has any magic attached (and after you update it), or you don't. So its all or nothing.


      Perl without its internal 'magic' implementation is essentially no longer Perl.

      Agreed. But I don't think the intention is for these to not work. Merely for a Perl implementation that is able to optimize particular sections of code that don't need magic. (Perhaps using a no magic; pragma as a hint to the compiler of what sections it can optimize.)

      That certainly seems a worthy thing to work on.

      use Moops; class Cow :rw { has name => (default => 'Ermintrude') }; say Cow->new->name

        The catch is that as soon as that section uses a parameter or a global, it has no chance to know whether there will be any magic!


        sub addOne { my ($num) = @_; return $num + 1; }

        need any magic? No? So what happens if I pass the subroutine a tie()d variable?

        I'd like to ask the OP to restrict himself to a recent version of English if he decides to reply. His attempts at old English are funny for two sentences, boring for ten and right away annoying for longer posts.)

        Enoch was right!
        Enjoy the last years of Rome.


        You are absolutely correct in every regard.

        People have been talking about a "no magic;" pragma for years:

        one more in support for "no magic" pragma, Anthony Baxter to Tom Christiansen, Tue 04 Mar 1997

        You will note, hopefully happily so, this is already codified in The Low-Magic Perl Commandments:

        LMPC #12. Thou Shalt Use Pragmata To Mark Magic Level Of Code (“use magic;” & “no magic;”)

        Thank you for your moral support! :-)

        ~ Will the Chill

      Thank you for your feedback, you win 1 brownie point for reminding me to ban weak references!

      I'm guessing you didn't actually read through the commandments, since everything you listed (except weak refs, now added to 21) is already covered:

      LMPC #21. Thou Shalt Not Use Typeglobs, Code References, Or Weak References

      LMPC #23. Thou Shalt Not Use Aliased, Non-Descriptive, Special (%ENV, %SIG, etc), Or Punctuation Variables ($_, @_, %!, $@, $$, $[, etc)

      LMPC #45. Thou Shalt Not Use Regular Expressions, Smart Match, q Quoting Mechanisms, Unicode/UTF-8, Or String Interpolation

      PBP #246. Don’t tie variables or filehandles.

      LMPC #35. Thou Shalt Not Use Taint Mode

      LMPC #15. Thou Shalt Not Use Deprecated (Array Value Exists, etc) Or Experimental Perl Features (Lvalue Subroutines, Attributes Pragma, etc)

      LMPC #17. Thou Shalt Not Use Dynamic-Type Data Or Auto-Vivification

      The RPerl compiler has 2 data modes: C/C++ ops & Perl data, and C/C++ ops & C/C++ data. When using the Perl data structures, we can actually keep the magic bits active if we want. Note, my use of the term "magic" refers to both the specific so-named magic bits attached to Perl data structures, as well as Perl's wacky/weird/complex operations in general.

      Also, we can use the "no magic;" pragma to turn off magic for 1 subroutine at a time, which means we can mix low-magic RPerl code (fast) with high-magic Perl code (maybe less fast), giving us the best of both worlds!

      Maybe you want to actually look at the things I wrote and perhaps even download some RPerl code?

      ~ Will
        Several of the things I listed are not in your commandments. For example you say "don't tie", but you don't say "don't use variables supplied by other people, which may or may not be tied", such as %Config. You say "don't use lvalue subs", but you don't warn against "substr($x,1,2) = $y".

        Maybe you want to actually look at the things I wrote and perhaps even download some RPerl code?

        I had a quick look, but frankly my strength was sapped by your previous incomprehensible Part 8 post.

        The thing I don't get (perhaps it's explained clearly somewhere, in which case I missed it), but what is the *point* of RPerl? Is it for making trivial blocks of perl code to run faster (but as soon as you want to do anything at all useful, you have to turn it off)?

        It seems to be named in honour of RPython, but that has a very specific, limited use case, and this seems unrelated.

        Also, I think the contribution of magic to making perl run slower is greatly overstated. The overhead that magic adds is a one-bit test with a conditional function call per variable access. There's still a big other bunch of overheads. For example polymorphic types. When you do something like

        $a + $b

        perl has to retrieve the the SVs from the current pad, check their 'get magic' flags and if set call mg_get() on them, then check whether overloading is enabled and if so call the overloaded add method, or failing that, try and convert their values into integers or floats, possibly doing a string to integer conversion, or stringifying a ref into "HASH(0x1234)" using the referent's address as an integer if it's a ref, or warning if the value is undef. It then does a whole bunch of stuff related to avoiding overflow or loss of precision where possible, such as upgrading from int to float or vice versa.

        Of all that, skipping the magic part is just skipping a 1-bit flag test - everything else still needs doing.

        (In fact, the actual check in pp_add() or's the flags of the two SVs at the top of stack together, and calls a function if the combined bits indicate magic or a ref: the latter indicating possible overload).



      Upon going back and re-reading this post of yours, I realize that you WIN 2 MORE BROWNIE POINTS! :)

      First, I needed to add $1 and $a to the list of banned special vars:

      LMPC #23. Thou Shalt Not Use Aliased, Non-Descriptive, Special ($1, $a, %ENV, %SIG, etc), Or Punctuation Variables ($_, @_, %!, $@, $$, etc)

      Second, even though tie() is banned in PBP, and I've tried my best not to repeat-ban items from PBP, I think your point is well-taken in that tie() is important enough to be double-banned and appear in both PBP & LMPC:

      LMPC #16. Thou Shalt Use A Type System (“use types;”), Static-Type Data, Variable Declarations, & Non-Tied Variables

      LMPC #17. Thou Shalt Not Use Dynamic-Type Data, Auto-Vivification, Or Tied Variables

      Thanks again!

      ~ Will
Re: Perl 5 Optimizing Compiler, Part 9: & The Low-Magic Perl Commandments
by Laurent_R (Canon) on Nov 25, 2013 at 23:16 UTC

    I do not know if this project is really going to go somewhere, but I definitely find some interest in it if I understand it correctly. I do not understand why it triggers such heated reactions, but possibly I am lacking some of the historical background.

    First, let me state clearly that I love what you call the "magics" of Perl and I am not willing to sacrifice that on the altar of speed, because this is the essence of Perl in my view, and that's one of the main reasons I love Perl. Speed is also very important to me, because I am dealing daily with pretty large amounts of data (dozens of GB), but, to me, making easy things easy and difficult things possible (whatever the exact phrase is...) is just more important. Most of the time, Perl is fast enough anyway for me (although, yes, I have sometimes to change my algorithm to get it to be fast enough). And there are a number of things in your "Low-Magic Perl Commandments" that I am not willing to accept. I want to be able to use tied data, code refs, closures, prototypes, etc.

    Having said that, when I have a severe performance problem, I would be quite happy to use a limited Perl subset if that solved my problem. From what you said, Will, if I understand correctly, it seems that it would be possible to have all the "magics" when needed, and apply the Commandments when speed really matters. Did I understand this right? If this is correct, then I think this is worth the trouble, the best of two words, I think you said somewhere. Given the "law" that 10% of the code usually accounts for 90% of the duration time, I would probably be willing to write those 10% in "low magics" Perl if that gave me a real performance increase when I need it, provided I can still use the high magics in the rest of my code. If this means that I can write this low magics 10% in limited Perl (rather than Inline C, XS or whatever), and can have the Perl that I love for the rest, then I think that I might be willing to buy your idea. Is my understanding correct or did I extrapolate too much on your comments?

    One last (but not least) point of concern. Your site states somewhere that RPerl is free for non commercial use. What does that mean exactly? Does that mean that companies would have to pay for using RPerl? If such is the case, then, sorry, I won't be with you, this is against my principles, I want these things to be free, in both senses of the word (free beer and free speech, as R. Stallman would say it, which does not mean that I agree with him on everything, far from that), and I am ready to donate money to make this possible (and i have done it quite a few times), but I certainly don't want to pay a license for using a programming language.

    Well, to conclude, the idea of your project looks promising in my view, and I would really be inclined to encourage you to go further, but I would need to know more to be able to make an educated opinion. Your site is often somewhat vague (your benchmark, for example, seems promising but some sentences ring some alarm bells in my mind, like the words "hand compiled", or something like that, used somewhere. What does that really mean? Is this an actual test of a product or even a prototype, or is this a "Proof of Concept" slide-ware for the sucker?

    Well, sorry if I am a bit harsh, I really don't mean to be nasty and I am really interested in your concept, but I would like to know more.

      Your site states somewhere that RPerl is free for non commercial use. What does that mean exactly...blahblah:)...

      Copyright © 2013, William N. Braswell, Jr.. All Rights Reserved. This work is Free & Open Source; you can redistribute it and/or modify it under the same terms as Perl 5.18.0.

      For licensing details, please see

        OK, thank you for the clarification. :-)
Re: Perl 5 Optimizing Compiler, Part 9: & The Low-Magic Perl Commandments
by einhverfr (Friar) on Nov 18, 2013 at 02:49 UTC

    First I don't want this to discourage you. I do think, however, that rperl would do better as a project aimed at expanding the realms in which Perl goes (i.e. aimed at new projects) than as a performance boost for existing projects. My current projects are entirely unlikely to ever use it, but hey if I find a use for it in a new project, I might consider it.

    I am however unimpressed by the assertion that the commandments necessarily lead to less buggy code. Almost every feature in Perl has a legitimate usage, and almost every feature can be horribly abused. If you don't believe that, I have a codebase to sell you... For example, properly used, Moo and Moose can lead to more maintainable, clearer, and hence less buggy code than the old low-level blessed hashref approach. Yes I have worked a lot with both. Similarly coderefs have some very important uses which require quite a lot of annoying efforts to get around if you want to get rid of them. Same with typeglobs, etc. In general working around a legitimate use for a given approach means that you end up with complicated, unclear code.

    Regarding creating clear, understandable code, the question very often times is when various things are elegant solutions to problems which do not otherwise admit of elegant solutions.


      You are correct, I expect a majority of projects utilizing RPerl will be new, rather than de-magicifying existing projects. The existing projects which are in critical need of a speed boost will have the option to upgrade their code to utilize the RPerl compiler.

      I never asserted that the commandments-as-a-whole lead to less buggy code. The only time the words "bug" or "buggy" appear are in commandments 4 & 5:

      LMPC #4. Thou Shalt Use Hard Work To Create Robust, Effective, Bug-Free, High-Quality Code That Serves A Real Need

      LMPC #5. Thou Shalt Not Use Laziness To Create Fragile, Ineffective, Buggy, Low-Quality Code That Just Saves Labor

      The exhortation to write non-buggy code is given in these 2 commandments, which is different than asserting one will write bug-free code by following all 64 commandments.

      Also, I never said that the commandments-as-a-whole lead to more readable code. The exhortation to write clear code is given in 4 more commandments, also in the "Ideals" section like the don't-write-buggy-code commandments above:

      LMPC #2. Thou Shalt Use Perl Best Practices (“perlcritic –brutal” & “perltidy -pbp”) & Idiomatic Programming

      LMPC #3. Thou Shalt Not Use Cleverness, Obfuscation, Golf, Or Non-PBP Code, Except As Officially Stated Herein

      LMPC #8. Thou Shalt Use Humility To Create Maintainable, Re-Grokkable Code That Serves The Greater Good

      LMPC #9. Thou Shalt Not Use Hubris To Create Unmaintainable, Incomprehensible Code That Just Impresses People

      So again, the idea of write-clear-code is given in the above 4 commandments, which is different than asserting one will write clearer code by following all 64 commandments.

      Which leads us to the important question: what is the point of The Low-Magic Perl Commandments, then???

      The purpose is to write low-magic code that can be made to run fast. Nothing more, nothing less.

      If any of the commandments are unclear, then please help me out so I can make them more grokkable.


      ~ Will the Chill
Re: Perl 5 Optimizing Compiler, Part 9: & The Low-Magic Perl Commandments
by ysth (Canon) on Nov 18, 2013 at 03:57 UTC
    From 2007-2009, there was a similar project "kurila"; you might want to look at some of the work there. Releases of kurila can be found on CPAN.
    A math joke: r = | |csc(θ)|+|sec(θ)| |-| |csc(θ)|-|sec(θ)| |

      Thanks for the info!

      I looked at kurila and found that, while it is also a Perl 11 type project like RPerl, our differences outweigh our similarities...

      From Kurila Intro POD:


      Experimenting with the Perl language.
      RPerl is only experimenting with speed, not new language features.

      Removing ugly things.
      Yes RPerl does remove some ugly parts of Perl 5.

      Adding ugly things.
      Heck no, RPerl isn't adding anything to Perl 5 other than speed, much less ugly things!

      Changing old design decisions.
      RPerl is only against slowness, not necessarily actual design decisions.

      Readable, modifiable perl source.
      Okay, sure, we all want more readable code, right? Right???

      Kurila is not intended to be compatible with Perl 5.
      RPerl compiles low-magic Perl 5 code into drop-in Perl-5-compatible replacement functions. Backward compatibility with Perl 5 and XS is important to RPerl.

      Interesting stuff!

      ~ Will the Chill
        although there are couple of unessential differences that you mentioned, there is one huge common thing between kurila and your rperl - these both have user base approaching to zero....

        read kurila ml archives
        the sooner you understand this the better, in terms of wasted efforts......

        speed is not the goal of perl, for this we have C and inline::c... perl is o(fun) and o(hard things easy) but not for speed.... you're mistargetting the goal, hence the aforementioned epic fail in the future......

Re: Perl 5 Optimizing Compiler, Part 9: & The Low-Magic Perl Commandments
by Will_the_Chill (Pilgrim) on Nov 10, 2013 at 17:30 UTC
Re: Perl 5 Optimizing Compiler, Part 9: & The Low-Magic Perl Commandments
by elTriberium (Friar) on Nov 12, 2013 at 00:03 UTC
    62. Use Low-Magic OO (Blessed Reference To Field Hash Or Restricted Hash PBP 195 Conflict)

    I'm out. I have zero interest in ever writing code that does that again.

      Howdy elTriberium,

      You definitely won't have to write any OO implementation code, RPerl provides it for you, just like Moose does. The difference is that RPerl only uses low-magic OO and Moose does high-magic OO. RPerl's low-magic OO can be compiled into C++ OO, Moose and other high-magic Perl OO not so much.

      The reason why this may have been confusing is because I do not mention RPerl itself anywhere in The Low-Magic Perl Commandments. This is because the LMPC are general guidelines to be followed by any low-magic Perl software, such as some of RPerl's sister projects under the Perl 11 umbrella, like Reini's P2 and Goccy's GPerl. So the LMPC don't require use of RPerl specifically, just low-magic strategies in general.

      Does that help clear things up?

      ~ Will the Chill

      I'm out. I have zero interest in ever writing code that does that again.

      And then what happened? The project has to start somewhere, and code that does that is how all OO in perl is built

Re: Perl 5 Optimizing Compiler, Part 9: & The Low-Magic Perl Commandments
by sundialsvc4 (Abbot) on Nov 12, 2013 at 03:03 UTC

    “Downvoted” though my comments were ... I at least did not lurk behind “Anonymous Monk” to say them.   (heh ...)

    Will, the root problem of your “grand pronouncement,” just like every graduate-student (or senior undergrad) who has “created a New Programming Language, accompanied by A Paper Extolling Its Superiority to Everything That Has Come Before,™” is simply that this really isn’t a terribly useful thing to have done.   Let me explain.

    “An implementation of the Perl-5 programming language,” on my present machine, “sub-optimal” though it may be, occupies all of about 86 kilobytes of object-code.   On the other hand, the CPAN library of the system is about 96 megabytes.   And the client project library, as previously mentioned, is more than a million lines of custom-written code which in fact cost well more than a million US Dollars to produce.   This situation is hardly atypical.

    Therefore ... in light of this, how fare your “commandments?”   Frank answer:   “not so good.”   Because they not only “demand” that I change, but “require” me to change ... and to reap what “benefits,” exactly?   Just in order to “Soak In The Goodness™ Of” whatever you have done.   Is this a cost that I can justify?   No, it isn’t.  

    Plan Accordingly.™

    Sure, “new code” is being written all the time, and so all of us are always clamoring for improvements.   But none of us can afford to de-stabilize, let alone change, “what has been done before.”   And this is very much what has made some of the truly-innovative “hacks,” like Moose, be so successful and widely-accepted.   It was “a significant improvement” that you could take advantage of with relatively low business risk. It’s not simply that you can use Moose;.   It’s every bit as much, if not much-more, that you can also say:   no Moose;.   You can mix the old with the new.

      Howdy sundialsvc4,

      I do applaud you for the non-anonymous remarks, even if I strongly disagree. :-) reap what "benefits", exactly?

      I thought by now the answer was obvious to everyone. YOUR CODE WILL RUN WAY FASTER.

      Is this a cost that I can justify?

      Sure, if you want YOUR CODE TO RUN WAY FASTER. No brainer.

      It was “a significant improvement” that you could take advantage of with relatively low business risk. You can mix the old with the new.

      Now you're just messing with me, right? I've explicitly and clearly stated this 3 times in this thread alone! Come on, man, get with it. To YET AGAIN quote myself:

      Also, we can use the "no magic;" pragma to turn off magic for 1 subroutine at a time, which means we can mix low-magic RPerl code (fast) with high-magic Perl code (maybe less fast), giving us the best of both worlds!

      As I've stated already, you can mix compiled low-magic code with normal non-compiled high-magic code.

      1. We can mix low-magic code with high-magic code.
      2. We can add back in all the high-magic components after we've got RPerl v1.0 working with the low-magic components.

      Simply and politely put, your arguments ARE TOTALLY INVALID.

      ~ Will
        Prove it. With real code.

        (Shrug...)   What can I say, Will?   It may astound you that I would say ... “I don’t particularly care that “my code will run ‘way faster!’ ”

        Really, the only thing that matters to me is that the software ... that is to say, the software that I have, and, that is to say, already ... produces the correct results, and that I can prove it.

        Furthermore, the actual extent of compiler-induced measurable improvements to the runtime of any code whatsoever is, at this point in time, a remarkably slight delta.   It is the stuff of “Whetstones” by now.   To make significant improvements to any compute algorithm, you have to change the algorithm.   Therefore, the usual actual business response is to “throw silicon at it (again),” thereby avoiding both the uncertainty and the expense of changing source, instead trusting Moore’s Law to once again save your business bacon.   After all, a capital investment in a $30,000 souped-up piece of silicon is nothing compared to the unfathomable expense of touching “paid-for source code that works.”

        The essence of your “commandments” is that “source-code change” is not only ‘free,’ it is ‘a Good Thing.™’ ”   Most unfortunately, this is not the case.   Aye, there’s the rub ...

        “Source-code change” ... is not an option.

        Think about it ... “why t’hell was Moose so successful, when it is a Wart on top of a Hack on top of a Bunion to try to make Perl-5 the object-oriented language that it never was, and never was intended to be?”   The short answer was that you could use Moose; to put on your Super Suit, and-d-d-d... that you could utter no Moose; to take it off again.   All of which enabled you to make Useful Improvements To™ “the garbage what you were stuck with” without “starting over.”   Sure, it was “inefficient,” but then again, “Chips Are Cheap.™”   And so, so it goes.

      An implementation of the Perl-5 programming language ... occupies all of about 86 kilobytes of object-code

      Ja ja ja...

      Hint: ldd is "your" friend.

      A reply falls below the community's threshold of quality. You may see it by logging in.
A reply falls below the community's threshold of quality. You may see it by logging in.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlmeditation [id://1061711]
Approved by kcott
Front-paged by ww
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others admiring the Monastery: (3)
As of 2024-06-22 21:59 GMT
Find Nodes?
    Voting Booth?

    No recent polls found

    erzuuli‥ 🛈The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.