Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Perl XS

by revendar (Novice)
on Nov 18, 2013 at 18:58 UTC ( #1063185=perlquestion: print w/replies, xml ) Need Help??
revendar has asked for the wisdom of the Perl Monks concerning the following question:

Hi. I'm new to Perl XS. I have worked on perl for years.But i have only basic knowledge in C. I'm trying to create a C program say test.c and inside it I have a print array function which accepts an array and loops and print. I have created a corresponding test.xs file. and .pm file. I have made and installed it. When I try to use it it works for char array , but when I print an int array.. it doesn't. Please find the code below

#!/usr/bin/env perl use ExtUtils::testlib; use Data::Dumper; use myClibs; print "Compiling C program...\n"; print qx{}; my $test = [ 1, 2, 3, 4, 5, 6, 9]; ArrayTest::print_array_char( "revendar" ); print "------------------------------------------------\n"; ArrayTest::print_array_char( @$test ); print "------------------------------------------------\n"; ArrayTest::print_array_int( "revendar",8 ); print "------------------------------------------------\n"; ArrayTest::print_array_int( @$test ,8 );

C prog is functions.c

#include <stdio.h> #include <string.h> #include <math.h> void print_array_char(char * array) { int l; int i; l=strlen(array); printf("Length of array is %d\n",l); for(i=0;i < l;i++) { printf("Element array[%d] = %c\n",i,array[i]); } } void print_array_int(const int array[], int l) { int i; for(i=0;i < l;i++) { printf("Element array[%d] = %d\n",i,array[i]); } }

xs file is myClibs.xs

#include "EXTERN.h" #include "perl.h" #include "XSUB.h" #include "ppport.h" MODULE = myClibs PACKAGE = ArrayTest PROTOTYPES: ENABLE #include <stdio.h> #include <string.h> #include <math.h> void print_array_char(array) char * array void print_array_int(array,l) char * array int l

pm file is

package myClibs; our $VERSION = '0.01'; require XSLoader; XSLoader::load('myClibs', $VERSION); 1;

My question is.. Can I not just write a complex C program and write a simple xs file containing the function/datatype names and compile it. Do I need to learn perlguts to achieve what I want. I'm trying to write a c program to replace a perl function which is slow and handles huge data. I am at the gates waiting for a solution/advice

Replies are listed 'Best First'.
Re: Perl XS
by ambrus (Abbot) on Nov 18, 2013 at 23:52 UTC

    The easiest way to work with XS is to put it in a full-blown perl module built with ExtUtils::MakeMaker. I'll show you how to do that.

    Meanwhile, we'll also have to fix your XS file, because the declaration of print_array_int doesn't make sense as is: XS won't know how to pass an array of ints to your print_array_int function. I've taken the liberty to rename and modify some other parts of your code too.

    First, create a directory which I'll call PrintArrays, this will be the directory where your perl module exists. You'll need five files in this directory.

    First, your library which does the heavy work, written in C, and a header for it. Write this to the file printfuncs.h to define the interface of your C code.

    #ifndef INCLUDE__PRINTARRAYS_PRINTFUNC__H #define INCLUDE__PRINTARRAYS_PRINTFUNC__H void print_array_char(const char *array); void print_array_int(const int array[], int l); #endif

    Put the implementation into printfuncs.c.

    #include <stdio.h> #include <string.h> #include "printfuncs.h" void print_array_char(const char *array) { int l; int i; l = strlen(array); printf("Length of char array is %d,\n", l); for (i = 0; i < l; i++) { printf("Element array[%d] = %c,\n", i, array[i]); } } void print_array_int(const int array[], int l) { int i; printf("Length of int array is %d,\n", l); for(i = 0; i < l; i++) { printf("Element array[%d] = %d,\n", i, array[i]); } }

    Now we have to write the Makefile.PL, which is the customary name for the script that calls ExtUtils::MakeMaker. Here, we'll ask MakeMaker to build your C code for you, with the same compiler as your perl is normally built. This has the advantage that your module is easy to deploy, because MakeMaker will automatically compile it together with the XS code, so you don't need an external dependency. Now if your C code were a large library used in places other than this perl module, you could of course compile it separately and require it as a separately installed dependency for this perl module, but we won't do that now. So write this to Makefile.PL

    use ExtUtils::MakeMaker; WriteMakefile( NAME => 'PrintArrays', VERSION_FROM => '', OBJECT => 'PrintArrays$(OBJ_EXT) printfuncs$(OBJ_EXT)', ); __END__

    Note the OBJECT clause which tells MakeMaker that printfuncs.c must be compiled to compile this module. This also refers to the XS part of the code which I'll show next.

    By default, MakeMaker will automatically scan your module directory (recursively) for .xs and .pm files, and knows how to build and install them automatically when you build and install the module. So our XS code will reside in a file called PrintArrays.xs. Here's how it starts (the easy part).

    #include "EXTERN.h" #include "perl.h" #include "XSUB.h" #include "printfuncs.h" MODULE = PrintArrays PACKAGE = PrintArrays PROTOTYPES: DISABLE void print_array_char(array) const char *array

    The print_array_char function will Just Work, because the XS compiler knows how to pass a perl string to a C function that expects a const char * argument. The print_array_int function is more complicated though, because it expects an array of C ints, which the XS compiler doesn't know how to create from your perl code. There are two ways around this, and I'll show the easy way first. We'll pack the integers to a C array in a wrapper function we write in perl, represented in a perl string, and pass that (plus a length) to an XS function called print_array_int2_impl. This makes the XS function quite simple. Append this to PrintArrays.xs

    void print_array_int2_impl(arraypacked, len) const char *arraypacked I32 len CODE: print_array_int((const int *)arraypacked, len);

    We then have to write the perl part of the code. This will do two things: load the XS extension, and provide the wrapper function called print_array_int2 that will do the above mentioned packing. The perl part of the code is in the file called and looks like this:

    package PrintArrays; use warnings; our $VERSION = "0.001"; use XSLoader; use Exporter 'import'; our @EXPORT_OK = qw(print_array_char print_array_int print_array_int2) +; XSLoader::load("PrintArrays"); sub print_array_int2 { my($arr) = @_; print_array_int2_impl(pack("i*", @$arr), 0+@$arr); } 1; __END__

    The XSLoader part loads the XS extension (this module is an alternative to Dynaloader, you can use whichever you prefer). The print_array_int2 function expects a single argument: a reference to an array of integers, which it will pack and pass to the XS function, which in turn calls the C function.

    Build this module by running perl Makefile.PL && make

    Now, if you wanted to use this module, you would install it by make install, which would copy both and shared the library built from the xs and C code to wherever perl modules normally reside in your perl installation. But now, we'll now just do a quick check in place. As the compiled module isn't yet installed to the normal location, but built in the ./blib directory, we'll have to tell perl to find it there. Run the following.

    perl -I./blib/lib -I./blib/arch -we 'use PrintArrays qw(print_array_ch +ar print_array_int2); print_array_char("grt"); $arr = [42,52,12,46,68 +]; print_array_int2($arr);'

    This should give the following output, all of which is printed directly from the C file.

    Length of char array is 3, Element array[0] = g, Element array[1] = r, Element array[2] = t, Length of int array is 5, Element array[0] = 42, Element array[1] = 52, Element array[2] = 12, Element array[3] = 46, Element array[4] = 68,

    I'll show a different solution for wrapping the perl_array_int C function under the fold.

    For my previous XS example on perlmonks, see also Re: Opposite of strtol?.

    Update: for docs on how XS works, see core perl documentation in perlxstut, perlguts, perlxs, perlapi, ExtUtils::MakeMaker.

      Excellent explanation. Thanks ambrus! Thanks Monks.
Re: Perl XS
by arkturuz (Curate) on Nov 18, 2013 at 19:31 UTC
    You don't have to write XS code at all. Maybe the better solution would be to use excellent Inline::C module.

    Also, post the slow Perl code, maybe we could optimize it a bit.

    Edit: better link to Inline::C added.

      Thanks!. I have seen Inline::C. I wanted to give XS a try. I still wanted to use XS as it keeps an abstraction.

        Inline::C can still be helpful - as a quick way of discovering what's going wrong and testing proposed fixes.

        For example, I placed (copy'n'paste') your 2 functions in an Inline::C script:
        use warnings; use strict; use Inline C => Config => BUILD_NOISY => 1; use Inline C => <<'EOC'; void print_array_char(char * array) { int l; int i; l=strlen(array); printf("Length of array is %d\n",l); for(i=0;i < l;i++) { printf("Element array[%d] = %c\n",i,array[i]); } } void print_array_int(int array[], int l) { int i; for(i=0;i < l;i++) { printf("Element array[%d] = %d\n",i,array[i]); } } EOC print_array_char( "revendar" ); print_array_int (-1,12,23,3);
        When I run that script, it compiles, then outputs:
        Length of array is 8 Element array[0] = r Element array[1] = e Element array[2] = v Element array[3] = e Element array[4] = n Element array[5] = d Element array[6] = a Element array[7] = r Undefined subroutine &main::print_array_int_alt called at line +....
        The problem is that, although there's nothing syntactically wrong with print_array_int(), perl doesn't know how to pass the 'int array[]' type to XS.
        For a working solution, you need to know a little bit about the perl API. I suggest perlxs, perlxstut, and perlapi docs, though you'll perhaps also find some useful tips in the Inline::C cookbook and, no doubt, many other places.
        Anyway, here's one solution:
        void print_array_int(int x, ...) { dXSARGS; int i; for(i=0;i < items - 1; i++) { printf("Element array[%d] = %d\n",i,SvIV(ST(i))); } XSRETURN(0); }
        "items" is the number of elements on the stack - so there's really no need to pass the length of the array to the function. You could remove that arg and rewrite the for loop condition as (i=0; i<items; i++)

        Two things to note about Inline::C:
        1) It's really just XS - it takes your C code, autogenerates the XS code, then compiles and runs your program.
        2) Inline::C defines its own stack macros, all of which begin with "Inline_Stack_" and are defined in the Inline.h file that it also autogenerates. Other than that, it's the same as XS, and you don't *have* to use its stack macros. You can just use the normal XS terms - as I did above when I declared "dXSARGS" instead of "Inline_Stack_Vars" (and used "XSRETURN(0)" instead of "Inline_Stack_Void").

Re: Perl XS
by Laurent_R (Abbot) on Nov 18, 2013 at 23:33 UTC

    Hmmm, if you were doing intensive CPU computation, then XS or Inline::c would definitely be good choice of candidates for improving performance. But if your performance issue is that you are handling huge amounts of data, I am not entirely sure that you are going to be easily doing much better than Perl. It really depends on the details of what you are doing, but Perl is fairly good at data munging, it is not always very easy to do any better even in a compiled language such as C. And, quite often, Perl offers you the tools to use easily a better algorithm which would be a pain in the neck to implement in C. BTW, this is not empty talk, I am working almost daily with tens of GB of data, often with hundreds of GB and sometimes (rarely, though) with even larger volumes, I have succeeded quite often (most of the time, actually) to create Perl programs faster or much faster than the existing pure C (or Java) programs they were replacing, just because Perl enabled me to implement easily better algorithms (sometimes, the speed of a Perl hash lookup can just make the hell of a difference). Of course, nothing in the foregoing is meant to claim that C is not better in some cases, I am only saying that the advantage of C is not always as obvious as you might think.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1063185]
Approved by Corion
Front-paged by kcott
NodeReaper settles into the armchair by the fire. You weren't using it were you?

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (6)
As of 2017-01-19 19:24 GMT
Find Nodes?
    Voting Booth?
    Do you watch meteor showers?

    Results (170 votes). Check out past polls.