http://www.perlmonks.org?node_id=853194

Preface

I know enough C and C++ to get things done, I'm definitely not a core Perl hacker, or any form of a low level programmer. With that in mind, there is a good chance I am incorrectly interpreting some of the terminology or the general approach. One area that I am very leary of is memory management. Just getting something working doesn't mean that you have all the leaks nailed down. That's my next task on this long, painful road. Oh, and before anyone decides to dump on my post, just keep in mind I've been trying to do this for a long time, and have bought books, scoured the internet, and asked people on forums. "This" was the BEST I could do.

So, I have always considered XS, the "final frontier" for me with Perl. I've been coding in Perl religiously for over a decade, and have done good bit of C and C++ as well. But marrying the two together has always been a formidable task that I have feared. I've avoided it through databases, sockets, and flat files. At my old job, there was an old crufty swig file in a project we used that miraculously was able to pull things together and provide an incredibly complex interface to a shared memory data store. I tried to make sense of it a few times, but in the end (due to a general lack of time and documentation) I ended up just trusting in the magic, and hoped that the swig code never broke.

I have built a few components in C++ that I am really proud of, and decided it was time to figure out how to link my beloved Perl to the world of C++. I like to think I am good at researching programming problems. I should really re-state this as, I am really good at using google, to solve compex programming problems. Sadly it's more like "I've never done anything unique, and anything I have problems with someone else has already had, and posted about it, years ago, and garnered responses from the luminaries that lead our community". I just am good enough with Google to find those threads to get me in the right direction to solve whatever problem I'm having, when my army of O'Reilley books aren't there to teach me what I need to know.

Here is my task. I have a C++ library, that I would like to be able to access from perl. The library is object oriented, takes strings as arguments, and returns strings, or vectors of strings, or hashes of strings/strings. I'll be honest, I'm not a C++ guru, I have used Boost and ACE to write safe multi-threaded code, and I have a decent understanding of memory management in C/C++ but that is pretty much the end for me.

BEGIN BOOK REVIEW

Disclaimer: This isn't a book review ;-), but I figured I'd make a few comments on this book I paid $44.99 for, since it's directly applicable to my problem.

To start off, I am staring at a copy of "Extending and Embedding Perl", by Tim Jenness and Simon Cozens. It is an odd book. Two of the eleven chapters are spent teaching you C, and two more chapters are spent discussing how to hack Perl itself. The rest of the book is a bit more applicable to the title, with two chapters devoted to embedding perl and five chapters to extending. (though, "Alternatives to XS" could have been a much larger topic, but I'll get to that later).

The reason I point out the distribution of material in the book is, in those 4 chapters about using XS, it misses the most basic use case in all of the examples. Linking Perl to an existing C or C++ library. Every single one of their examples shows the C code being embedded in the ".xs" file directly. There is no mention of how you would even accomplish this, anywhere in the book! Just to prove the point, go download the example code yourself and do an "find . |egrep -i '\.ch$'". You won't find anything outside of chapters one and three, the ones that deal with teaching you C, and a short blib on h2xs in chapter 7, but even that shows you the actual C code in the ".xs" file.

There are 7 pages devoted to extending to C++. I typed in the examples, and with a little work, got them working. The example didn't cover external files, nor did it cover how to pass complex variables (the examples dealt with "int's" for input an output). So for, to summarize (finally), this is a good reference book, and when I actually need to jerk around with SV's and HV's, and IV's, then I can learn what they are and how they work.

If I could change it, I'd just get rid of all the extra stuff, and make more room for XS and Perl's internals. If there was room to spare, maybe full chapters on Swig and Inline, as they are both solid solutions that deserve more than short mentions.

END BOOK REVIEW

Thus begun my internet search.

I started at Perlmonks. Queries concerning: "XS C++" and "XS external library" were fruitless for the most part. http://perlmonks.org/?node_id=517931 .. tye, had the most impressive response (yes, I upvoted him), but it only dealt with C++ loaded directly in the XS file, and with "char's".

Then I hit the chatterbox. tye of course was online, being the helpful guy he is held an amazing amount of patience with me as I slowly answered my questions. He did his absolute best to steer me away from XS entirely, and to use SWIG or Inline::C++ instead. But my position was simple, I didn't see any of the "awesome" libraries on CPAN resorting to this, why should I? Why is this so damn hard? Why is it so poorly documented? Why do I have a book on the subject that I paid 50$ for and I still have no clue how to accomplish such simple tasks? tye sensed my frusrtration, and kept pressing for alternatives. It was at this point I decided that I'd play the "Picard" card with XS.

<bold>"They invade our space, and we fall back. They assimilate countless worlds, and we fall back. Not again. Not this time. The line must be drawn here! This far, no farther! And I will make them pay for what they have done!" - Jean Luc Picard, Star Trek First Contact</bold>

Come hell or high water (?) I am going to figure this out.


Inline::C++ : This module, when it works, works really well. It provides a seemless integration between Perl and C++, and you barely have to lift a finger. The people who wrote this are beyond smart. I applaud their work. Here is where it breaks down for me. You need to have a compiler and your environment set up the same way for every machine that you want to use your library on (unless you do even more smart env logic in your BEGIN blocks). If you don't have a compiler, and/or don't want to rely on re-compiling when you run the program, I'm told you can distribute the compiled bits, but packaging this became almost as much of a daunting task as the original problem itself.

SWIG : I used to work with a guy who worked on the SWIG team. His initials are "M.M" (for friends of mine that read this). Nobody can understand a word he says (in English), as he's from Chile, but what is funny is nobody can understand what he says in spanish eiher (and that comes from people who are native spanish speakers). Heh, anyway, that's sort of how SWIG is. You tell it to do something, and either it works, or it spits back unintelligble mess that is totally undecipherable. Just like my old job, you have to trust the magic, and if it breaks, you have no recourse. Their website has decent documentation, but not enough detail on how to resolve complex issues, leaving you to scour the internet like I did on XS. Do you use an abstraction that will have problems, but a smaller pool of people that can help? Or do you tough it out and use the harder solution, but one that has a larger pool of support? Guess which one I'm going with.


I tried to just look at existing perl modules that I knew linked to C/C++ on CPAN. I knew libxml, Wx, and the various DBD::* libraries had to make calls to the underlying layers, but the typemaps were far too complex for my puny mind to comprehend. I hoped the Wx library would have some insight, but it too blew my mind. I needed a concrete example or documentation on what steps needed to be taken.

John Keiser seems almost as frustrated as I am. This link is referenced in MANY other articles on the internet concerning XS and C++. Of course, this doesn't have an example of using external files, and ::sigh:: it only deals with int's for inputs and outputs. Sadly, since this was written in 2001, I ran into some issues just getting the examples working. In the end, I just gave up on it and looked for other documentation.

Stack Overflow is a pretty good place to get answers to questions, most of the answers pointed to SWIG, Inline::C, and the perl-xs-tut (we'll get to the xs-tut here in a second).

Perl XS Mailing list 1 and 2 : I found these gems, while digging, and it gave me hope that building a typemap for std::string and std::vector might be possible. Unfortunately, the code was meant more as a scratchpad (and he states this explicitly). So close!

The Perl XS mailing list is a really great site. This place has a great signal to noise ratio (similar to the mod_perl mailing list). Yet again, things that I have questioned, more or less, have already been asked, answered, or told they are thinking about the problem wrong (the latter being my problem).

Boom!, here was some documentation of an external file with XS. Lots of concepts and little example though. I really wanted everything explained, no more magic.

WrappingPerl : Uses external files! Uses the STL!! ... Uses SWIG. Damn it. Ok, so I typed all this out, and got it working, and tried to figure out what SWIG was generating based on the resulting ".cxx" file. It ended up being another brick wall. I was impressed that SWIG was capable of doing almost everything I needed it to do, but it was the same magic that I had to "trust" like I did back in my previous job. I decided that I was NOT going to use swig.

Kaye SWIG : An even better SWIG example. It was recently posted to Alberto Simões's perl blog by Kaye (courtesy of Shlomo Yona). This is the best SWIG documentation that deals with complex data structures and external files available. If anyone else writes a book about XS, they should just dump this entire distribution to paper, as it is incredibly well documented. Too bad it's C, otherwise I probably would have given up XS and just used this.

Perl XS Tutorial : Yes! Why the hell didn't I look here first?! Example 4 is perfect! External files! MYEXTLIB! (wtf is an MYEXTLIB?!)

As an aside, look at the perldoc for ExtUtils::Makemaker, under MYEXTLIB, and see if you make the connection that THIS is the attribute that allows you to reference external libraries, yes I know EXTLIB should have clued me in, but it didn't register as the solution to my problem).

Holy crap the ExtUtils::MakeMaker is complicated. Where the heck did this "MY" namespace come from? What is all this weird "make" snippets littered all around? Ok, just get it working, and then you can reverse engineer it so you can teach yourself what all these keywords mean so you can actually learn something, rather than trusting the "magic". So that's what I did. (I'll cover that when I finally get to the example).

I used Module::Install for my last CPAN projet, "Ravenel". I wondered if it could build XS modules as well. But it's funny to me, since everyone seems to hate ExtUtils::MakeMaker, but all the modules that use XS all use it. It looks like this is the only syste worth using if you're dealing with this stuff.

Ok, let's not get too excited, I still need to know how to build a typemap for STL based objects, and if it's possible, complex ones (Vectors and Maps).

Typically, in any google search of a technical nature, you inevitable will start seeing posts in languages other than your own, I don't start hitting these until I exhaust all of the ones that are in english (obviously). (What do you call someone who is in Europe and can speak 5 languages? A waiter. What do you call someone who speaks one language and who is in Europe? An American.) Google translate does a decent job of getting the point across, but the concepts backing the idea always seem to be lost in translation (love that movie).

Japanese blog : I ran into this first. And I thought I finally found it. The typemaps that wrap the STL for Perl. I have a "simple" class that you instantiate with a string as an arugment, and some methods that take strings and spit them back out. This is formatted for SWIG, and combined with the earlier lesson I found on swig, I probably could have just got by with this. But I've come too far now. XS or bust!

Holly's Blog But, this, is when I hit the proverbial jackpot. "Holly's page". I don't know who Holly is, but he/she is amazing and I wish I could sing his or her praises from the tallest buildings / trees / mountains. Holly has a 9 part XS C++ programming course on his or her site "The introduction perlxs 1-9". It covers damn near everything I would ever want to know about XS, with concrete examples.

Dear Holly, you rock, you have my undying devotion until the end of time, because you have documented your brilliance with so much clarity that I was able to translate it from Japanese and still get what I needed out of it. I owe you a beer, sake, fruit juice, or whatever it is you drink. Thank you from the bottom of my beaten down soul. --dextius

I feel better now that I've gotten that diatribe out of the way. Let's show you an example of using an external library, passing strings back and forth between Perl and C++. Then, let's explain what every line does so we can demystify some of the magic and fear that comes when dealing with XS. Before I dump out all my code, there is one point I need to make. I found it's easier to abandon your build script that you normally use to build your C++ project, and just use what Makefile.PL generates. I accomplished this by simply copying my code into this new directory tree.

Here is the code

Change directories in your shell to somewhere where you can create new stuff and then run this.

h2xs -A -n SimpleTest

This will create a skeleton for us to work with. Now create the SimpleLib directory.

mkdir SimpleTest/SimpleLib

Here is the code for our "simple" library.

/SimpleTest/SimpleLib/Simple.h

#include <string> #include <map> #include <vector> using namespace std; typedef map<string,string> StringMap; typedef StringMap::iterator StringMapIt; typedef vector<string> StringVector; typedef StringVector::iterator StringVectorIt; class Simple { public: Simple(int myArg); int add(int myArg); std::string get_string(std::string str); StringMap getMap(StringMap myMap); StringVector getVec(StringVector myVec); protected: int myInt; };

/SimpleTest/SimpleLib/Simple.c

#include "Simple.H" #include <string> Simple::Simple(int myArg) { myInt = myArg; return; } int Simple::add( int myArg ) { return myInt + myArg; } std::string Simple::get_string( std::string str ) { std::string foo = "|" + str + "|"; return foo; } StringMap Simple::getMap(StringMap myMap) { myMap["Hello"] = "world"; return myMap; } StringVector Simple::getVec(StringVector myVec) { myVec.push_back("abc"); myVec.push_back("123"); return myVec; }

/SimpleTest/SimpleLib/Makefile.PL

use ExtUtils::MakeMaker; ( $CC, $Verbose ) = ( 'g++', 1 ); # define g++ as your CPP compiler WriteMakefile( NAME => 'Simple::SimpleLib', # the name of the library we'r +e building SKIP => [qw(all static static_lib dynamic dynamic_lib)], clean => {'FILES' => 'libSimpleLib$(LIB_EXT) unitTests'}, # +when you run make clean, it'll delete the shared library and the bina +ry too. CC => $CC, # Our CPP compiler LD => '$(CC)', CCFLAGS => '-fPIC', # this instructs the CPP compiler to use " +Position independence", basically a shared library, more details here # http://www.fpx.de/fp/Software/tcl-c++/tc +l-c++.html ); # Ok, this "MY::top_targets" command isn't even mentioned in the pod f +or ExtUtils::MakeMaker. # Just understand that it is a method that allows you to modify the ta +rgets that the makefile will build. # (when you type just "make", you're really invoking "make all"). # You can see that "all" points to static, which points to libSimpleLi +b.a, which actually does the real work. # O_FILES refers to the object files that are created when the ".c" fi +les are compiled. # You can see what is going on here when you actually run "make" # g++ -c -fPIC -O2 -march=nocona -mmmx -msse -msse2 -msse3 -DVERSI +ON=\"\" -DXS_VERSION=\"\" Simple.C # g++ -c -fPIC -O2 -march=nocona -mmmx -msse -msse2 -msse3 -DVERSI +ON=\"\" -DXS_VERSION=\"\" unitTests.C # ar cr libSimpleLib.a Simple.o unitTests.o # : libSimpleLib.a # "ar" is an archiving program that takes object files and packages th +em into archives. It reminds me a lot of tar, except minus the raw d +evice related stuff. # When you run "make bin" it takes the object files created when we ra +n "make" and links them to create the binary unitTests (since unitTes +ts has a "main" function defined). # g++ Simple.o unitTests.o -o unitTests sub MY::top_targets { ' all :: static pure_all :: static static :: libSimpleLib$(LIB_EXT) # These must be indented with tabs!!! libSimpleLib$(LIB_EXT): $(O_FILES) $(AR) cr libSimpleLib$(LIB_EXT) $(O_FILES) $(RANLIB) libSimpleLib$(LIB_EXT) # Tab indent these lines bin: $(O_FILES) $(LD) $(O_FILES) -o unitTests '; }

/SimpleTest/SimpleLib/unitTests.c (A quick test "main")

#include "Simple.H" #include <iostream> #include <string> int main( int argc, char * argv[] ) { Simple s = Simple(5); std::cout << s.add(9) << std::endl; std::string moo = "abc"; std::string foo = s.get_string(moo); std::cout << foo << std::endl; StringMap m; m["qqq"] = "uuu"; StringMap n = s.getMap(m); std::cout << "n[qqq] " << n["qqq"] << std::endl; StringVector v; v.push_back("999"); StringVector w = s.getVec(v); std::cout << "w[0] " << w[0] << std::endl; std::cout << "w[1] " << w[1] << std::endl; std::cout << "w[2] " << w[2] << std::endl; return 0; }

At this point, you can compile "SimpleLib". You'll run these commands.

perl Makefile.PL make make bin

To verify, run the unitTests binary.

./unitTests

It should return:

14 |abc|

Before I go any further, lets look at the Makefile.PL as it is key to all of our future plans.

1. We need to build a shared library, "make" will generate the "libSimpleLib.a" file.

2. "make bin" will generate the unitTests binary, running a "file unitTests" will tell you what architecture you're building for.

3. It is important your version of Perl was compiled with the same architecture (perl -v will tell you this).

4. A great resource on make is available here: Makefile Tutorial

Ok. Great, now we have a shared library built, and trust that it actually is capable of doing something when told to (via the unitTests binary). Now let's hook this up to Perl.

/SimpleTest/Makefile.PL

use 5.008003; use ExtUtils::MakeMaker; $CC = 'g++'; # See lib/ExtUtils/MakeMaker.pm for details of how to influence # the contents of the Makefile that is written. WriteMakefile( NAME => 'Simple', VERSION_FROM => 'lib/Simple.pm', # finds $VERSION PREREQ_PM => {}, # e.g., Module::Name => 1.1 CC => $CC, LD => '$(CC)', ($] >= 5.005 ? ## Add these new keywords supported since 5.005 (ABSTRACT_FROM => 'lib/Simple.pm', # retrieve abstract from mod +ule AUTHOR => 'dextius <dextius@blahblah.com>') : ()), LIBS => [''], # e.g., '-lm' DEFINE => '', # e.g., '-DHAVE_SOMETHING' INC => '-I.', # e.g., '-I. -I/usr/include/other' # OBJECT => '$(O_FILES)', # link all the C files too # +Un-comment this if you add C files to link with later: # target the shared library we just created "MYEXTLIB" tells make +to use this shared library. 'MYEXTLIB' => 'SimpleLib/libSimpleLib$(LIB_EXT)', ); # (THE INDENTION on "cd SimpleLib..." MUST BE A TAB) sub MY::postamble { ' $(MYEXTLIB): SimpleLib/Makefile cd SimpleLib && $(MAKE) $(PASSTHRU) '; }

/SimpleTest/Simple.xs

# Ahh now we are getting to the meat and potatoes of this monster. # Most of this mess is generated by h2xs, but let's see what else we g +ot. #include "EXTERN.h" #include "perl.h" #include "XSUB.h" #include "ppport.h" # This line being ABOVE the "MODULE =" line is critical, as everything + above that line is treated as pure C++ # Anything below it is in XS macro land. #include "SimpleLib/Simple.h" using namespace std; MODULE = Simple PACKAGE = Simple # The constructor. Kind of looks like C++ here. I pass in an "int". Simple * Simple::new( y ) int y # I have no idea how "SimplePtr" became a valid keyword, but somehow t +he XS engine realizes that it is the return type of the constructor. # It's magic to me, for now. MODULE = Simple PACKAGE = SimplePtr # Here is some basic XS, the documentation on this is covered pretty w +ell in "Extending and Embedding Perl", specifically chapters 2 and 6. int Simple::add(x) int x OUTPUT: RETVAL string Simple::get_string(z) string z OUTPUT: RETVAL StringMap Simple::getMap(myMap) StringMap myMap OUTPUT: RETVAL StringVector Simple::getVec(myVec) StringVector myVec OUTPUT: RETVAL

typemap

TYPEMAP Simple * T_PTROBJ string T_STRING StringMap T_STRING_MAP StringVector T_STRING_VECTOR # Ok, let's step through this one fragment at a time. First we're goi +ng to declare the "input" type # (ie: an argument passed in from perl to C++) of T_STRING, which maps + to the C++ STL type of std::string. # # SvTYPE is a function that looks at an SV's type to determine if it's + an ordinary SV. # It's basically the XS version of "ref". SVt_PV is a string. But yo +u should use SvROK (the other half of "ref") # first, not sure why it's not. If the string typemap receives someth +ing that isn't one, it'll warn and return undef. # SvCUR is a function that lets you interrogate the length of a scalar +. # SvCUR has a friend named SvLEN that tells you how long it could be. # If it's length is 0, warn and return undef (not always desired, but +for now it's fine. # The last line generates a string from the argument given. # "string" is the actual C++ class constructor. # SvPV_nolen lets you extract the value of a scalar and coerce it as a + char *, # of unlimited length, because the C++ std::string constructor doesn't + care, I think. INPUT T_STRING { if (SvTYPE($arg) != SVt_PV) { warn(\"${Package}::$func_name() -- $var is invalid svtype\ +"); XSRETURN_UNDEF; } if (SvCUR($arg) == 0) { warn(\"${Package}::$func_name() -- $var is empty\"); XSRETURN_UNDEF; } $var = string(SvPV_nolen($arg)); } # Ok, this next section covers strings returned from C++ functions or +methods (hence the OUTPUT), # that need to be transformed back into scalars for perl to deal with. + The "sv_setpvn" macro sets # the value of a scalar. OUTPUT T_STRING sv_setpvn($arg, $var.c_str(), $var.size()); # Now things get complicated, let's handle an "array reference" and tu +rn it into a std::vector of std::string's # and pass it into a C++ function or method, yee-hah.. # # AV *av is a perl array. # I32 len is a 32 bit int. (why not just use int?) # SvROK "is this a reference" sort of like ref, except it doesn't tell + you which kind. (SvRV dereferences the reference) # SVt_PVAV is Perl's C structure for an Array. # So, the entire line says, "if the argument is a perl array reference +:" # Dereference the arg, then downcast it to a pointer of an array type +(since we know it is one), assign that to our array type (av). # Assign the length of the arraa, using "av_len". # We return undef if it's length is 0 (not sure why we couldn't return + an empty list here, but whatever) # The "else" handles the case when it's not handed an array reference +as an argument # # Now for the guts. Iterate over every element (the for loop)..Call t +he vector's method "push_back". # Create a new string, calling the scalar extraction SvPV_nolen, again +st the dereferenced value of what av_fetch returned. # It passes the array, the position, and a third argument detailing if + Perl needs to auto extend the list # (because you're about to write to that index position). av_fetch re +turns a pointer to a pointer to an SV, hence the deref'ing "*av_fetch +". # lastly, we need to assign the "$var" (what goes to C++ to t_sv, our +C++ StringVector, the header shows this is a std::vector<std::string> INPUT T_STRING_VECTOR { AV *av; I32 len; StringVector t_sv; if(SvROK($arg) && SvTYPE(SvRV($arg)) == SVt_PVAV){ av = (AV *)SvRV($arg); len = av_len(av) + 1; if(len == 0){ warn(\"${Package}::$func_name() -- $var is empty array + reference\"); XSRETURN_UNDEF; } } else { warn(\"${Package}::$func_name() -- $var is not a array ref +erence\"); XSRETURN_UNDEF; } for (I32 i = 0; i < len; i++) { t_sv.push_back(string(SvPV_nolen(*av_fetch(av, i, 0)))); } $var = t_sv; } # Now for String vectors returned back to Perl # If it the C++ vector is empty, it returns undef back to perl # Why is it when I see "sv_2mortal" I think about the intro to the Seg +a game, "Altered Beast"... "Rise from your grave!" # So many weirdly named keywords in Perl. I love the language, maybe +even because of it's quirks. # # Anyway, we create a new Array. (the rest is covered similarly on pa +ge 124 of Extending and Embedding Perl). # # SvSetSV will COPY the data from one variable to another, unless they +'re pointing to the same thing. # newRV_noinc: This is the C/C++ equivalent of the "\" or reference op +erator. We have a list, we want to return it as an array reference. # We don't increment because it needs to go out of scope when this blo +ck finishes (we're copying to "arg" anyway, so the data lives on). # Reading from inside out. Take an array, cast it as a pointer to a s +calar (what newRV_noinc requires). Pass that to newRV_noinc to # create a reference. "arg" and this new reference value get sent to +SvSetSV to copy the contents of your new array ref to the outbound # "arg" structure. Phew! OUTPUT T_STRING_VECTOR { if($var.empty()){ warn(\"${Package}::$func_name() -- vector is empty\"); XSRETURN_UNDEF; } AV *av = (AV *)sv_2mortal((SV *)newAV()); for(StringVectorIt it = $var.begin(); it != $var.end(); it++) +{ av_push(av, newSVpvn(it->c_str(), it->size())); } SvSetSV($arg, newRV_noinc((SV *)av)); } # Now we need to look at hash referenecs passed as arguments to C++ fu +nctions. # They'll transform into std::map<std::string, std::string>. # HV is a hash value # HE is a hash entry (a key) # Ok, we've seen most of this already. So let's move quick. # If it's not a hash ref, the return undef (SVt_PVHV is a hash) # Next is the C side perl loop over the keys in a hash # The key and value from the hash get forced into scalar values via He +SVKEY_force and HeVAL. # Those scalars are passed into the insert function of our StringMap ( +std::map<std::string, std::string>) # Assign the ingoing $var to the StringMap we've been inserting into! INPUT T_STRING_MAP { HV *hv; HE *he; StringMap t_sm; if(SvROK($arg) && SvTYPE(SvRV($arg)) == SVt_PVHV) { hv = (HV *)SvRV($arg); if(hv_iterinit(hv) == 0) { warn(\"${Package}::$func_name() -- $var is empty hash +reference\"); XSRETURN_UNDEF; } } else { warn(\"${Package}::$func_name() -- $var is not a hash refe +rence\"); XSRETURN_UNDEF; } while((he = hv_iternext(hv)) != NULL) { SV *svkey = HeSVKEY_force(he); SV *svval = HeVAL(he); t_sm.insert(StringMap::value_type(string(SvPV_nolen(svkey) +), string(SvPV_nolen(svval)))); } $var = t_sm; } # Lastly, deal with functions / methods that return StringMap's back t +o Perl. # If it's empty, return undef.. # That crazy 'rise from the dead' thing again creating a Hash value th +at will go out of scope and be freed after this block ends. # Now it's basic C++ map looping. hv_store only takes 5 arguments. : +:blink:: # The hash that you want insert into # A const char* of the key # An I32 integer of the length of that char* key we just mention +ed # A scalar that is to be the value of the key. # The hashing value (passing 0 will tell Perl to compute it for +you) # it->first is the key of the current element in std::map # it->second is the value of the current element in std::map # The value needs to be "wrapped" by the creation of a new scalar valu +e, same as what we did when adding elements to an array OUTPUT T_STRING_MAP { if($var.empty()){ warn(\"${Package}::$func_name() -- map is empty\"); XSRETURN_UNDEF; } HV *hv = (HV *)sv_2mortal((SV *)newHV()); for(StringMapIt it = $var.begin(); it != $var.end(); it++) { hv_store(hv, (it->first).c_str(), (it->first).size(), newS +Vpvn((it->second).c_str(), (it->second).size()), 0); } SvSetSV($arg, newRV_noinc((SV *)hv)); }

What do I do when I get: "error: macro "do_open" requires 7 arguments, but only 2 given"

While I'm here, I ran to a bunch of other fun issues. Linking to Boost or ACE causes all kinds of wacky symbol collisions to fire off, but google was to the rescue, again. Simple add these few lines to your ".xs" file (immediately after #include "ppport.h")

#undef init_tm #undef do_open #undef do_close #ifdef ENTER #undef ENTER #endif

Exceptions? So, my code throws exceptions. I cannot catch them, they cause Perl to blow up. I looked around, and eventually found the link above. It, like most of what I have learned about XS frightened me. I can see this is another "spend a week" researching on how other people have approached the problem, and then hack something of my own together, but hopefully someone will read this and tell me what the best approach is.

Multiple C++ objects exposed by XS (in a single project)?

Wx does this, but it uses it's own custom MakeMaker to pull it off. The one thing I noticed is that it created seperate .xs files for each class. I assume this is what I need to do, but I don't know how to make Makefile.PL pick up the other .xs files I create describing the other objects defined in my header file. I also learned (the hard way) you can't define more than one object in a ".xs" file, or maybe it's just the syntax I was using. I really don't know. I wish I had a few weeks to reverse engineer how Wx is doing this, but there is so *MUCH* code, I'm not even sure where to start.

Summary

Wow, this was really hard. I still don't know everything. I feel like after pouring over Extending and Embedding Perl over the last month or so I can tackle a C based library with some level of competency. But beyond these simple STL calls and an object wrapper, I still feel totally lost with C++ and XS. So many burning questions, and so few outlets for more information. Again, if I didn't stated it before, the fact that I got any of this working at all wouldn't have been possible with out "Holly's Blog". It's amazing how much information is on that page. It's ironic, that I had to go through Japanese to to get to XS (both of which being undecipherable to me).

What have I learned? That C++ is just another opaque object to Perl, that the typemaps are what allow you to call internal perl5 api's to construct perl datastructures, using C++ code right along side it. Once you learn it, it makes sense, but holy moly, what a hill you have to climb to get there. I also learned that I have a lot more to learn.

Hopefully this mess I have written will inspire much more talented people to write even better documentation so I can update this entire page with a simple link to their page.

Thanks to all the people who wrote these posts on their blogs, and to Perlmonks, and to tye for "inspiring" this :-)

  • Comment on Interfacing Perl with C++, using XS, with external files, and using the STL as parameters and return values.
  • Select or Download Code

Replies are listed 'Best First'.
Re: Interfacing Perl with C++, using XS, with external files, and using the STL as parameters and return values.
by mojotoad (Monsignor) on Aug 05, 2010 at 21:07 UTC
    Your tenacity is to be admired. Thanks for documenting your quest.

    Recently I pointed out the perl-xs implementation of google protocol buffers to tye. I dumped my scratch pad into a post: Protocol Buffers and XS

    Perhaps you'll be interested to see how that project autogenerates XS around C++.

    Cheers,
    Matt

      Hahah, you did that for tye, because I was in chatterbox bugging him, and you happened to be around. I lost your link during the chaos, damn. This would have been helpful. What I find interesting is you directly reference object files via the "OBJECT" attribute in your Makefile.PL. Where I used the "MYEXTLIB" directive. I am pretty sure I understand the difference. Also, XSOPT? I know it's a TMTOWTDI world, but you appear to be taking a totally different approach to get the same thing accomplished. Wow.
Re: Interfacing Perl with C++, using XS, with external files, and using the STL as parameters and return values.
by BrowserUk (Pope) on Aug 05, 2010 at 19:10 UTC

    Congratulations! Both for persevering in your research and goals; and for getting all that down.

    I'd love to see that broken into 2 or 3 sub-posts and indexed from the root.

    The rant/history is valuable--and fun to read; especially as I've been part way down some of those dead ends myself previously--but having the tutorial part broken out as one or two smaller posts for easy reference would be very useful.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      I would break it apart, but I hate the thought of polluting perlmonks with multiple posts. Do you think just cutting the code sections out to another post would be ok?

        I don't think it would be "pollution".

        Personally, I'd break out the code sections and related narrative and post them as replies to the root and add links in the root to them.

        An (IMO: poorer) alternative, would be to wrap chunks of the post in <readmore></readmore> tags.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Interfacing Perl with C++, using XS, with external files, and using the STL as parameters and return values.
by BrowserUk (Pope) on Aug 06, 2010 at 09:20 UTC
    Exceptions? So, my code throws exceptions. I cannot catch them, they cause Perl to blow up.

    FWIW: In my version of Devel::Size.xs, I catch exceptions. The method I use works for MSVC & gcc on both 32 & 64-bit.

    Whether it is compatible with C++ I don't know.

      Wow. I love Devel::Size, I've used that module to test production code many times. Thank you so much for such a useful module. I'm pretty sure my approach is incompatible with exceptions, as I do all my work in the typemap, and I use the XS to directly map to my target function. What I "should" be doing is creating an insulating class, that can catch the exception at the XS layer, and then rethrowing the perl exception from there. This is basically what you are doing in your code. I still think this is unfortunate. You'd think there would be some handy macro that would handle this sort of thing for you.
        Wow. I love Devel::Size, I've used that module to test production code many times. Thank you so much for such a useful module.

        The qudos for Devel::Size firmly belongs to the original author, Dan Sugalski. I just tweaked it a bit to make it a little faster, more reliable and use less memory.

Re: Interfacing Perl with C++, using XS, with external files, and using the STL as parameters and return values.
by TGI (Parson) on Aug 06, 2010 at 05:46 UTC

    Amazing post, great detective work. Thank you so much for sharing.

    Have you tried asking advice on the WxPerl mailing list? The Wx devs are very smart and helpful bunch.


    TGI says moo

      I didn't know there was a mailing list. I guess I should try that, but I always feel bad bugging people for help. Hopefully they'll understand that I will update this monster post with their knowledge if they give me some help with my issues :)
Re: Interfacing Perl with C++, using XS, with external files, and using the STL as parameters and return values.
by tsee (Curate) on Aug 15, 2010 at 09:47 UTC

    For some reason, you entirely missed a tool called "XS++" which started its existence as the Perl-C++ glue for Wx.

    Please have a good look, it will most certainly make your life easier wrt. wrapping C++.

    References:

      Awesome, I was hoping I'd get a hold of someone who was on top of this.

      The fact that I missed XS++, is pretty laughable, and definitely NOT my fault. I must have re-worded 'perl "C++" XS' 50 different ways on multiple search engines and never ONCE hit XS++. That is pretty frustrating, considering the time and effort I put into researching this, AND the time that Mattia and you must have spent writing XS++ and all this documentation. Heck, I even emailed the Wx mailing list asking for pointers on handling exceptions and multiple objects. I INCLUDED a link to this very post. Mattia himself didn't mention that I wasn't using ExtUtils::XSpp!

      NOBODY mentioned anything about XS++ to what I was doing, not my book (Extending and Embedding), not anyone on Perlmonks, not Ironman Perl, not google. Obviously, I've never heard of it, and OBVIOUSLY, nobody ELSE has either.

      This whole process just feels like a kick in the teeth, and this is the cherry on top.

        I suppose you're talking about the thread on wxperl-users starting here? Quoting Mattia's reply with added emphasis:

        At the moment there is partial support for handling exceptions in XS++: by default all exceptions are caught, and the XS wrapper die()s with a message. If the exception is derived from std::exception, the message includes the text of the what() method, otherwise a generic error message is thrown. Support for propagating the complete exception object to Perl is planned but not implemented yet.

        If for any reason you don't want to use XS++, you need to put a try/catch block around the XS code and handle the exceptions manually

        So while the mention of XS++ could have been more prominent, it was at least there. After all, you were asking this on the wxperl mailing list, so I guess Mattia assumed that you at least had a glance at the source code.

        Either way, Extending and Embedding Perl couldn't have mentioned it. It's an old book. The first standalone release of ExtUtils::XSpp was just a year ago. There are not that many distributions on CPAN that use it and the tools that make it more convenient to use are still in their early stages. The slides of the talk that I pointed you at were from this year's YAPC::EU which ended less than two weeks ago. All things consdidered, it's not that surprising XS++ didn't pop up as the first thing when searching for XS and C++.

        If you're using XS++ and like it, then there's something you can do about the visibility problem yourself: Blog about it. Tell people that and how you use it. Tell us how it helped you or, even better, how it could be more useful. We won't exactly send you away if you wanted to get involved with code either.

Re: Interfacing Perl with C++, using XS, with external files, and using the STL as parameters and return values.
by deMize (Monk) on Aug 05, 2010 at 23:11 UTC
    zomg that's a beast of a post. I'll let you know what I think, when I'm done with it next year.


    Demize