Re: Perl 5 Optimizing Compiler, Part 9: RPerl.org & The Low-Magic Perl Commandments

[dave_the_m]

Perl without its internal 'magic' implementation is essentially no longer Perl.

Here's some of the features of perl that would no longer work in sections of code where magic was disabled:

$1, $2, et al, and in fact most other special variables, such as  $., $!, %ENV, %SIG, @ISA etc;

while (/.../g) (/g match in scalar context);

weak references;

ties; so for example this stops working: use Config; if ($Config{foo}) ...

the taint mechanism;

$#array, keys %hash, substr, vec in lvalue context;

foo($hash{nonexist}), where the element is only vivified if foo() assigns a value to $_[0];

It would also interfere with offset caching on utf8 strings, meaning that for example //g on long strings will go quadratic on length.

And of course if you write a function which is allowed to be called by others, then you have to allow that @_ may contain magic values which won't work properly if you've disabled magic in your function. Similarly, the return value you receive from other functions or perl builtins and ops may contain magic.

In short, all but the most trivial of perl code requires magic to be enabled.

Note also that you can't have just 'low' amounts of magic; in a particular section of code you either perform a check on each SV before you access it to see if it has any magic attached (and after you update it), or you don't. So its all or nothing.

Dave.

[tobyink]

Perl without its internal 'magic' implementation is essentially no longer Perl.

Agreed. But I don't think the intention is for these to not work. Merely for a Perl implementation that is able to optimize particular sections of code that don't need magic. (Perhaps using a no magic; pragma as a hint to the compiler of what sections it can optimize.)

That certainly seems a worthy thing to work on.

[Jenda]

The catch is that as soon as that section uses a parameter or a global, it has no chance to know whether there will be any magic!

Does

sub addOne {
  my ($num) = @_;
  return $num + 1;
}

need any magic? No? So what happens if I pass the subroutine a tie()d variable?

I'd like to ask the OP to restrict himself to a recent version of English if he decides to reply. His attempts at old English are funny for two sentences, boring for ten and right away annoying for longer posts.)

[Will_the_Chill]

Hi Jenda,

My only use of old English is The Book Of RPerl, and perhaps to a trivial degree the use of "thou shalt" and "thou shalt not" in The Low-Magic Perl Commandments. Just for you, just in this post, I'll replace "Thou Shalt Not" with "DON'T". :-)

To address your concerns, globals are not allowed and tied variables are not allowed.

LMPC #17. DON'T Use Dynamic-Type Data, Auto-Vivification, Or Tied Variables

LMPC #19. DON'T Use Private Dynamic-Scope (“local”), Private Persistent Lexical-Scope (“state”), Global, Or Package Variables (“our”)

Does that answer your questions?

Thanks,
~ Will

[Will_the_Chill]

tobyink,

You are absolutely correct in every regard.

People have been talking about a "no magic;" pragma for years:

one more in support for "no magic" pragma, Anthony Baxter to Tom Christiansen, Tue 04 Mar 1997

You will note, hopefully happily so, this is already codified in The Low-Magic Perl Commandments:

LMPC #12. Thou Shalt Use Pragmata To Mark Magic Level Of Code (“use magic;” & “no magic;”)

Thank you for your moral support! :-)

Perling,
~ Will the Chill

[Will_the_Chill]

Dave,

Thank you for your feedback, you win 1 brownie point for reminding me to ban weak references!

I'm guessing you didn't actually read through the commandments, since everything you listed (except weak refs, now added to 21) is already covered:

LMPC #21. Thou Shalt Not Use Typeglobs, Code References, Or Weak References

LMPC #23. Thou Shalt Not Use Aliased, Non-Descriptive, Special (%ENV, %SIG, etc), Or Punctuation Variables ($_, @_, %!, $@, $$, $[, etc)

LMPC #45. Thou Shalt Not Use Regular Expressions, Smart Match, q Quoting Mechanisms, Unicode/UTF-8, Or String Interpolation

PBP #246. Don’t tie variables or filehandles.

LMPC #35. Thou Shalt Not Use Taint Mode

LMPC #15. Thou Shalt Not Use Deprecated (Array Value Exists, etc) Or Experimental Perl Features (Lvalue Subroutines, Attributes Pragma, etc)

LMPC #17. Thou Shalt Not Use Dynamic-Type Data Or Auto-Vivification

The RPerl compiler has 2 data modes: C/C++ ops & Perl data, and C/C++ ops & C/C++ data. When using the Perl data structures, we can actually keep the magic bits active if we want. Note, my use of the term "magic" refers to both the specific so-named magic bits attached to Perl data structures, as well as Perl's wacky/weird/complex operations in general.

Also, we can use the "no magic;" pragma to turn off magic for 1 subroutine at a time, which means we can mix low-magic RPerl code (fast) with high-magic Perl code (maybe less fast), giving us the best of both worlds!

Maybe you want to actually look at the things I wrote and perhaps even download some RPerl code?

Thanks,
~ Will

[dave_the_m]

Several of the things I listed are not in your commandments. For example you say "don't tie", but you don't say "don't use variables supplied by other people, which may or may not be tied", such as %Config. You say "don't use lvalue subs", but you don't warn against "substr($x,1,2) = $y".

Maybe you want to actually look at the things I wrote and perhaps even download some RPerl code?

I had a quick look, but frankly my strength was sapped by your previous incomprehensible Part 8 post.

The thing I don't get (perhaps it's explained clearly somewhere, in which case I missed it), but what is the *point* of RPerl? Is it for making trivial blocks of perl code to run faster (but as soon as you want to do anything at all useful, you have to turn it off)?

It seems to be named in honour of RPython, but that has a very specific, limited use case, and this seems unrelated.

Also, I think the contribution of magic to making perl run slower is greatly overstated. The overhead that magic adds is a one-bit test with a conditional function call per variable access. There's still a big other bunch of overheads. For example polymorphic types. When you do something like

$a + $b

perl has to retrieve the the SVs from the current pad, check their 'get magic' flags and if set call mg_get() on them, then check whether overloading is enabled and if so call the overloaded add method, or failing that, try and convert their values into integers or floats, possibly doing a string to integer conversion, or stringifying a ref into "HASH(0x1234)" using the referent's address as an integer if it's a ref, or warning if the value is undef. It then does a whole bunch of stuff related to avoiding overflow or loss of precision where possible, such as upgrading from int to float or vice versa.

Of all that, skipping the magic part is just skipping a 1-bit flag test - everything else still needs doing.

(In fact, the actual check in pp_add() or's the flags of the two SVs at the top of stack together, and calls a function if the combined bits indicate magic or a ref: the latter indicating possible overload).

Dave.

[Will_the_Chill]

Dave,

You win another brownie point! I did not strongly enough ban non-variable non-slice lvalues, the commandments have been updated yet again thanks to your sharp eye!

LMPC #20. Thou Shalt Use Scalars, Arrays, Hashes, Filehandles, & Mundane Lvalues (Variables & Slices)

LMPC #21. Thou Shalt Not Use Typeglobs, Code References, Weak References, Or Magic Lvalues (Builtins, Non-Variables, Non-Slices, etc)

I must disagree with your comment about not-using-other-people's-maybe-tied-vars, commandment #3 says not to use non-PBP code, which includes code-with-tied-vars written by either you _or_ somebody else.

LMPC #3. Thou Shalt Not Use ... Non-PBP Code ...

My previous post (Perl 5 Optimizing Compiler, Part 8: The Book Of RPerl) was made over 6 weeks ago, surely that's been enough time for your strength to return? ;-)

I think your questions about RPerl are generally answered in The RPerl FAQ, clearly linked at the very top of the Part 9 original post. Still, to answer your question specifically, the point of RPerl is to create an optimizing Perl 5 compiler. To achieve this, RPerl will initially remove all magic from Perl 5 and create a "restricted" (like RPython) subset of the Perl 5 language which can be compiled directly to C/C++ code, which is itself 100% compatible with, and equivalent to, the original Perl 5 code. This is implemented in RPerl by generating specially-crafted C/C++ code that can be fed through Inline::C(PP) and tie back into Perl 5 via XS. Once we have RPerl working with low-magic Perl 5 code, we can start selectively adding back in the magic components 1-by-1, retaining the ability to turn off all magic at any time to keep the pure speed boost of low-magic compiled code. As I've stated already, you can mix compiled low-magic code with normal non-compiled high-magic code.

To quote myself from my most immediately previous response to you, "Note, my use of the term 'magic' refers to both the specific so-named magic bits attached to Perl data structures, as well as Perl's wacky/weird/complex operations in general." You're right in pointing out that the magic bits attached to the Perl data structures are only 1 part of the "magic" that needs to be turned off to achieve massive performance benefits. This is why there are more commandments under the "Operations" section than the "Data" section.

Perling,
~ Will

[Will_the_Chill]

Dave,

Upon going back and re-reading this post of yours, I realize that you WIN 2 MORE BROWNIE POINTS! :)

First, I needed to add $1 and $a to the list of banned special vars:

LMPC #23. Thou Shalt Not Use Aliased, Non-Descriptive, Special ($1, $a, %ENV, %SIG, etc), Or Punctuation Variables ($_, @_, %!, $@, $$, etc)

Second, even though tie() is banned in PBP, and I've tried my best not to repeat-ban items from PBP, I think your point is well-taken in that tie() is important enough to be double-banned and appear in both PBP & LMPC:

LMPC #16. Thou Shalt Use A Type System (“use types;”), Static-Type Data, Variable Declarations, & Non-Tied Variables

LMPC #17. Thou Shalt Not Use Dynamic-Type Data, Auto-Vivification, Or Tied Variables

Thanks again!

Perling,
~ Will