Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

XS Modules - why and when?

by Bod (Parson)
on Dec 05, 2023 at 00:22 UTC ( [id://11156077] : perlquestion . print w/replies, xml ) Need Help??

Bod has asked for the wisdom of the Perl Monks concerning the following question:

I know little about XS modules but I was looking at RPi::DHT11 and noticed that it is an XS module which seemed a little strange. Hence the question here. I was going to email stevieb, the author and ask but thought it would be good to get other people's take on this.

My understanding of an XS module is that it uses XSLoader to load C / C++ / C# code. The Perl is just a wrapper around the loaded code with the Perl doing little processing. I believe the main reason to do this is for speed as compiled languages of the C family are faster than interpreted languages such as Perl. Is that about right so far?

Assuming that is something like correct...
Why would the RPi::DHT11 module use XS? The DHT11 is a temperature and humidity sensor with a maximum frequency of 0.5Hz (one read per 2 seconds). So speed is not the issue here!

What reason would there be for creating an XS module in this use case? Would it not be easier and simpler and just as fast given the speed of the sensor, to write all the code in Perl?

I'm hoping for more insights into the world of XS modules here...

Replies are listed 'Best First'.
Re: XS Modules - why and when?
by LanX (Saint) on Dec 05, 2023 at 00:41 UTC
    > is for speed as compiled languages of the C family are faster than interpreted languages such as Perl. Is that about right so far?

    No, it's primarily about interfacing with an existing C library.

    Speed is only one possible reason for using/creating that library.

    Another is connecting to an API without native Perl support.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    see Wikisyntax for the Monastery

      Compatibility with defined behaviour of variables in another programming language (overflow etc) is another reason, like recently discussed in Trying to translate overflowing JS code to Perl. It might just be easier to load an existing library or interpreter than to recreate all that behaviour in Perl.

      Another popular reason is security. I'd rather trust some XS bindings to the always updated and audited OpenSSL library, written by security experts, used by some of the biggest companies in the world) to secure my connections than i would to some pure-perl reimplementation made by a one-person team in their spare time.

      Plus, why go to all the hassle of designing a complicated set of Perl modules to, say, implement an interpreter/interface/protocol/crypto library when some nice chaps on the internet have already gone through all the hard parts and gave you a perfectly fine C library with all the trimmings and bugfixes and workarounds. All you have to do is write a nice XS wrapper(*) around it and you're done. Unless the developers of that lib decide to change the interface, you even get all the updates for free ;-)

      (*) If you're lucky and the interface is simple enough, h2xs might even do a lot of the work for you. Thumbs up for Perl scripts that writes Perl scripts for you!

      PerlMonks XP is useless? Not anymore: XPD - Do more with your PerlMonks XP
      No, it's primarily about interfacing with an existing C library

      Thanks...that makes a lot of sense. cavac's point about security resonates here -> Re^2: XS Modules - why and when?

      But in RPi::DHT11, there doesn't seem to be any reason to use XS unless the C code already existed. I had assumed stevieb wrote the C code as well as the Perl code, but I have nothing to base that on.

      If we look at RPi::PIGPIO::Device::DHT22 we can see a pure Perl method of reading the output from the device (DHT11 and DHT22 are read the same way). As Perl is rather good at bitwise and mathematical operations, it just seems weird to me that XS would be used in this situation.

        I quickly scanned the POD for RPi::DHT11 and found:

        > This module requires the wiringPi library to be installed

        And http://wiringpi.com/ states:

        > WiringPi is a PIN based GPIO access library written in C

        And

        > is intended for use by experienced C/C++ programmers.

        So, doesn't this answer your question?

        Cheers Rolf
        (addicted to the Perl Programming Language :)
        see Wikisyntax for the Monastery

        If we look at RPi::PIGPIO::Device::DHT22 we can see a pure Perl method of reading the output from the device (DHT11 and DHT22 are read the same way).

        Ahhh, but wait! Understand that the library you mention is a TCP connector to a daemon that is running on the Pi hardware. That daemon is what does all of the GPIO register manipulation directly. That daemon is written in guess what? C! So the library you mention does not interact with the hardware directly. It communicates with software that does. My software directly interfaces the hardware, which is why I needed C/XS to do so. This library communicates with someone elses software via a TCP socket that does the down and dirty work.

        See here for info on pigpio.

        From the pigpio web page:

        The pigpio library is written in the C programming language. The pigpio daemon offers a socket and pipe interface to the underlying C library.

        So that's why the library you mention can be written in Pure Perl. Note that the author of the Perl library you mention and the author of pigpio are not the same person.

Re: XS Modules - why and when? (XS References)
by eyepopslikeamosquito (Archbishop) on Dec 05, 2023 at 01:14 UTC

    Bod, surprisingly I don't seem to have a list of references on this topic yet. This looks like a good place to start:

    See also:

    Other cool references welcome.

    👁️🍾👍🦟
      Great, the WP article is already detailing 5 common reasons: (emphasis added)

      > It may be desirable for a Perl program to invoke a C subroutine in order to handle very CPU or memory intensive tasks , to interface with hardware or low-level system facilities , or to make use of existing C subroutine libraries.

      Cheers Rolf
      (addicted to the Perl Programming Language :)
      see Wikisyntax for the Monastery

      surprisingly I don't seem to have a list of references on this topic yet

      Sorry to add to your workload!

Re: XS Modules - why and when?
by eyepopslikeamosquito (Archbishop) on Dec 05, 2023 at 05:44 UTC

    > My understanding of an XS module is that it uses XSLoader to load C / C++ / C# code

    Good luck with C# :) Despite its name, C# is much closer to Java than C (there's even a Comparison of C# and Java wikipedia page). Historically, C# was invented by Microsoft after years of expensive legal battles with Sun over Java licensing ... so they could hardly call it Java# :) - see this node for more details.

    Note that the XS (Perl) wikipedia reference states:

    XS is a Perl foreign function interface through which a program can call a C or C++ subroutine

    Though XS is essentially C-based, it's possible to provide a C interface to C++ code (BTW, the original 1983 Cfront C++ compiler converted C++ to C).

    👁️🍾👍🦟
      Good luck with C# :)

      I had assumed, perhaps wrongly, that the XS mechanism allows Perl to be connected to any compiled language, at least in theory... So it could be used with Ada or Zig or anything in between.

        Not directly, no. From perlxs:

        XS is an interface description file format used to create an extension interface between Perl and C code (or a C library) which one wishes to use with Perl.

        ...

        The glue code pulls the arguments from the Perl stack, converts these Perl values to the formats expected by a C function, calls this C function, and then transfers the return values of the C function back to Perl.

        So XS is inextricably tied to C. That is not to say that you could not use XS via C as a conduit to access other, non-C code but it would be a bit Heath Robinson. Perhaps FFI::Platypus would be a better bet for that.

        FTAOD, I am in no way an XS expert and can count on the fingers of one hand the number of times I have written such code so treat all this with a little scepticism.


        🦛

        I had assumed, perhaps wrongly, that the XS mechanism allows Perl to be connected to any compiled language, at least in theory...

        Unfortunately not. As someone already mentioned, it has the capability to play nice with C++ code (wiringPi is in fact C++ based), but there are a lot of gotchas that can prevent it from working, and often requires overriding/rewriting the functions that don't work in pure C before they can be presented through the Perl library.

        I also write a fair amount of C# code (berrybrew would be my most complex example), but wrapping something like that with XS would be an impossible, futile, headdesking task.

        XS is always C, but can call into anything that exposes C-compatible functions in a shared library. So, C works, and C++ works as long as the author of the C++ module exposes plain-old-functions with C calling convention. There are other languages like Rust that can expose C functions from a compiled library even though the library was generated from Rust code. In fact, quite a lot of compiled languages can expose C library APIs just because C is sort of like a common meeting ground for interoperation and language designers want their modules to be widely usable. But XS can't just call into arbitrary other languages on their home turf.

        Also I'd hesitate to call C# a compiled language. It's more like a bytecode interpreted language with optional just-in-time compilation. I don't think it can generate .so or .dll files with C calling convention. (but I'm too lazy to go find out)

        > I had assumed, perhaps wrongly, that the XS mechanism allows Perl to be connected to any compiled language, at least in theory

        While I'm not an XS expert -- and further to ++hippo's excellent reply -- I'd put both Java and C# in a separate category due to their reliance on an underlying virtual machine.

        I'm not aware of any plans to implement Perl 5 on the JVM (Java) or the CLR (C#). That this would require a huge technical effort is indicated at:

        Update: from the FFI::Platypus docs:

        Things not supported include languages that do not compile to machine code ... like .NET based languages and Java

        👁️🍾👍🦟
Re: XS Modules - why and when?
by stevieb (Canon) on Dec 05, 2023 at 18:07 UTC

    Sometimes I use XS to wrap existing C libraries. WiringPi::API is probably one of my larger examples. Not only do I wrap all of WiringPi with it, but within the XS, I often override or wrap the wrapped C functions because I need something to work slightly differently. here is one example.

    Other times, I interface with ICs and other devices where clocking bits in and out with precise timing is extremely critical to proper operation (example). That timing I found is often more reliable when written in a compiled language and then presented with a Perl interface.

    Other times it's purely for speed for sure.

    Sometimes, I've already written something in C for a project that I want to break out into a standalone project, so I put the C code into XS, then present it as a Perl interface. In these cases, I often will also write a Pure Perl port of it. Bit::Manip is one example. That distribution is purely written in C. The XS portion simply presents the C functions to the Perl library.

    Sometimes I use XS for academic purposes. I don't write a whole lot of C code so I like to dabble in it from time to time before it fades from my memory completely. Personally I don't write the XS, I write C, then have Inline::C develop the XS for me.

    In the case of RPi::DHT11, one has to turn high/low physical GPIO pins to read the data from the sensor. Using the wiringPi library allowed me to do that with an existing interface so I didn't have to write my own. That's why I spent two years wrapping the wiringPi C library in near its entirety. Writing something like that in pure Perl would have been impossible to deal with such low level hardware registers. The timing wasn't the issue here; the manipulation of the GPIO hardware registers was.

    -stevieb

Re: XS Modules - why and when?
by Marshall (Canon) on Dec 05, 2023 at 00:43 UTC
    Raw speed is just one consideration. In this case, I suspect that there is C language Unix device driver for the Raspberry Pi and that the XS module is the "glue" between that driver and Perl. The PIO driver will probably have to do some low-level stuff that is not well-suited to Perl and perhaps not even possible, like perhaps writing to a specific hardware address. So XS as an interface layer to the C language device driver is completely plausible to me.

    I am sure that stevieb can tell us the details of this implementation.

Re: XS Modules - why and when?
by davido (Cardinal) on Dec 05, 2023 at 22:26 UTC

    Another reason some things are done in XS is out of convenience in getting Perl to do things that Perl isn't made to do. There are modules written that mess around inside of Perl's internals in ways that would be really hard to do, if not nearly impossible to do, in Perl code.


    Dave

Re: XS Modules - why and when?
by stevieb (Canon) on Dec 05, 2023 at 18:41 UTC

    I forgot another example of why someone may use C/XS that I've had experience with. One of my clients a few years ago had a Perl system that they deployed at client sites. The software was proprietary (ie. non-Open Source), but as we all know, there's no real way to obfuscate it so it's easily prone to theft and copying.

    My client wanted a way to prevent this easy theft, so I suggested that we write some of the critical core components in C, compile it, then present it to Perl. That way, it's beyond trivial for the software to be stolen. Sure, the client could see all of the Perl code, but we didn't include the C library source, just the compiled shared library. Not perfect by any stretch, but pretty effective.

Re: XS Modules - why and when?
by stevieb (Canon) on Dec 05, 2023 at 20:48 UTC
    The DHT11 is a temperature and humidity sensor with a maximum frequency of 0.5Hz (one read per 2 seconds). So speed is not the issue here!

    Yes, data can be read only once every two seconds, but that's not the timing one needs to be concerned with. The real timing issue is in issuing digital signals to the device to inform it that we're about to read data from it. This is measured in microseconds. Perl, at best can reliably deal in milliseconds.

    You can extract two formats of data from this device. First, you have to hold the signal pin LOW for 18us (microseconds) to tell it you want to read data, then set it back HIGH. For the first format that is returned, the device sets the data pin low for exactly 50us (microseconds), then immediately hold it high for 26-28us. For format two, holds low for 50us, then high for 70us. If you're off even marginally sending or receiving these signals, you'll possibly get the incorrect format, corrupt or no data at all.

    So on a Raspberry Pi, even while using C, you can still get mis-timed events because of possible contention of the CPU. This is why microcontrollers are so much better for this. They operate instructions in a linear fashion and never have the overhead of other processes competing for CPU cycles.

    Also, Perl can't manipulate hardware registers like this directly. Even if it could, the timing would be too unreliable to be effective.

Re: XS Modules - why and when?
by stevieb (Canon) on Dec 20, 2023 at 09:41 UTC

    To bookend this whole thing, I want to say that I would stay away from the DHT* type sensors.

    For almost 10 years, I had a significantly nice multi-stage indoor cannabis growing operation that I used them for to help me with my fully automated hydroponic system.

    This is but my personal experience. Not only had I written software against them for the Pi, I also have software that I wrote for my real controllers, little Node MCU units, particularly the ESP-8266 and its derivatives (I still use frequently my WeMOS D1 Mini units). The DHT units required a lot of averaging and mediating of numbers to ensure a correct result. I learned by using them though. I learned a lot about math. Monks here taught me about "filtered moving average" and "probability normalization" and such. That's a fun game, literally, until I just want to code more.

    I don't like to give advice unless asked, but I would say that instead of the DHT* units, based on my experience with the difficulties they presented me, it may be prudent to investigate other options on hardware for the rather delicate situation** that faces you.

    -stevieb

    ** Inserting devices and custom software into your HVAC may influence how your insurance company looks at you, so I would definitely consider this as delicate.