Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much

Refactoring Perl5 with XS++

by rje (Deacon)
on Apr 25, 2015 at 05:06 UTC ( [id://1124642] : perlmeditation . print w/replies, xml ) Need Help??

Last time I mused aloud about "refactoring" Perl, I referenced Chromatic's statement/challenge:

"If I were to implement a language now, I'd write a very minimal core suitable for bootstrapping. ... Think of a handful of ops. Think very low level. (Think something a little higher than the universal Turing machine and the lambda calculus and maybe a little bit more VMmy than a good Forth implementation, and you have it.) If you've come up with something that can replace XS, stop. You're there. Do not continue. That's what you need." (Chromatic, January 2013)

I know next to nothing about XS, so I started reading perldoc.

I'm thinking about the problem, so if there is a question, it would be "what NOW?"

Should I bother with thinking about bytecodes? In what sense could it be a replacement for XS? What does "replace XS" even MEAN? (i.e. perhaps it just means "remove the need to use perl's guts to write extensions, and improve the API").

Most importantly, am I wasting people's time by asking?

I'm trying to come up with my own answers, and learn by trying. But wisdom is in knowing that some of you guys have already thought through this. If you can help bring me up to speed, I'd appreciate it.

UPDATE: I see even within the Lorito page, it was understood that the discussion was to some degree about Perl's API: "This is an issue of API design. If we understand the non-essential capabilities we want to support (e.g. optimization passes, etc), we can design the API so that such capabilities can be exploited but not required. - cotto "

Replies are listed 'Best First'.
Re: Refactoring Perl5 with XS++
by ikegami (Patriarch) on Apr 27, 2015 at 01:38 UTC

    XS is means of exposing C functions to Perl programs. This can be quite trivial, but you can also include C code to handle more complex types or to provide a more Perlish interface. A simple XS module:

    #define PERL_NO_GET_CONTEXT #include "EXTERN.h" #include "perl.h" #include "XSUB.h" #include <math.h> MODULE = MyModule PACKAGE = MyModule double sin(double x)
    package MyModule; use strict; use warnings; use Exporter qw( import ); use DynaLoader qw( ); our @EXPORT_OK = qw( sin ); bootstrap __PACKAGE__; 1;

    That said, I suspect Chromatic was referring to the Perl API rather than XS. (And even then, probably to a subset of it.) The Perl API allows you to create Perl variables and manipulate them. You also have access to Perl's stack, and you can call Perl subs. (XS code can use the Perl API.)

      Maybe so. Re-reading his post, though, it looks like he has wanted something that replaces XS, as well.

      He lays out three steps. First, a minimal bootstrapping language which can replace XS. Second, Rakudo's intermediate structure to represent the language (or something better). Third, Rakudo's metaprogramming system (or something better).

      To hammer the point home, he then says "Figuring out a replacement for XS is essential".

      And in another place, he actually tried writing a little bytecode interpreter which granted the power of C, without having to write in C. I think that's one peep into his vision, in a way.

        The fact that the document is about replacing XS doesn't change what I said. XS is no more a programming language than HTML is. The ops his bootstrapping language would have would have parallels in the Perl API, not XS.

        I think a pertinent question, then, is replace XS how? In one sense, XS is easily replaced if one has a powerful enough foreign function interface (FFI). In another sense, syntax that allows (possibly inlined) C to be separately compiled and to call back into Perl easily is required. If one wants to fully replace XS, the language has to be able to do both or do equivalent things to both.

Re: Refactoring Perl5 with XS++
by Anonymous Monk on Apr 25, 2015 at 07:12 UTC

      Same person?

Re: Refactoring Perl5 with XS++
by stevieb (Canon) on Apr 25, 2015 at 07:06 UTC

    Quoting chromatic is a big deal (IMHO). Are you saying you've "come up with something"?

    There was no question asked, so I'll ask one... are you thinking of a new implementation of 'something'?


      I'm thinking about the problem, so if there is a question, it would be "what NOW?"

      Should I bother with byte codes? What's a useful minimal set of operations? In other words, not just a bare Turing machine. In what sense could it be a replacement for XS? Is a forth-like stack machine going to create headaches later?

      Most importantly, am I wasting people's time by asking?

      I'm trying to come up with my own answers, and learn by trying. My first attempt used the Lua ops in an interpreter prototype. Of course all my prototyping is likely going to start with Perl, because I like it.

      But I'm also thinking about Perl's functions, and what it would take to "write" them in a lower level opcode set, rather than C.

Re: Refactoring Perl5 with XS++ (moved)
by LanX (Saint) on Apr 25, 2015 at 12:12 UTC
    As pointed out already, no question asked.

    Moved directly from SOPW to Meditations, because consideration process already blocked with reap request.

    Cheers Rolf
    (addicted to the Perl Programming Language and ☆☆☆☆ :)
    Je suis Charlie!

Re: Refactoring Perl5 with XS++
by sundialsvc4 (Abbot) on Apr 28, 2015 at 01:24 UTC

    There is also the more-philosophical matter of ... “why do we use XS?’   I think that a careful examination of that question will lead to the IMHO that Perl5’s way probably does make sense.

    Basically, we use XS because of the “80/20 Rule.”   80% of the execution time is spent in 20% of the code.   Therefore, 80% of the code is not particularly time-critical, therefore we have lots of more-or-less equally good options for how to build those things.   Writing those things in a high-level language and running it under an interpreter is “more efficient for us,” so we do most things that way.   Which leaves:   XS.

    The interpreter, in its lugubrious way, will hand off control to the jackrabbit, which eventually will hand the car-keys (and a result) back to the interpreter.   Any API that does for the jackrabbit what the jackrabbit needs, without slowing the jackrabbit down too much, is all that is required.   We don’t need fancy-pants.   We just need amazingly-cool C/C++ code in the hands of a really great compiler.

    And the internal design of the interpreter doesn’t matter that much, either.   Bytecodes or parse-tree, take your pick.   That portion of the overall program will probably spend most of its clock-time in some wait state or another, spending milliseconds(!) asleep, so it actually doesn’t matter too much (within reason) how fast it runs when it’s awake.   “Refactoring” that part probably won’t pay-back the effort spent to do it.

      JavaScript optimization proves you wrong.

        It is amazing what can now be done with JavaScript ... perhaps one of the most-studied language implementations in the world ... and the case for pure-C/C++ coding is definitely shrinking thereby.   But there is still a gap.   And, many of the “optimizations” that are being done in the JS world effectively consist of the clever application of XS behind-the-scenes.   Tools like Cordova make such things more explicit, but there is still an “external function interface” lurking behind.   Even so, it is often used to “get directly to the hardware” [of a mobile this-or-that].   The number of cases where “the slight overhead of an interpreter can’t be tolerated” is quickly shrinking, for a number of reasons, both hardware and software.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlmeditation [id://1124642]
Approved by stevieb
Front-paged by stevieb
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (6)
As of 2024-02-28 16:20 GMT
Voting Booth?
My favourite way to spend a leap day ...

Results (26 votes). Check out past polls.