Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Dealing with XS modules on multiple platforms

by Mutant (Priest)
on Jul 24, 2008 at 08:45 UTC ( #699808=perlquestion: print w/ replies, xml ) Need Help??
Mutant has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I'm trying to put together a process at work for building and deployment of CPAN modules. The idea is that when someone decides they need to use a CPAN module, they download it, build it, and add it to the repository of our deployment tool. The deployment tool will then copy those CPAN libraries to various boxes in various environments as needed.

This is fine (and fairly simple) for Pure Perl modules. The trouble comes with XS modules, because we're running 2 or 3 different platforms (all variations of Solaris). Obviously, these modules need to be built for each platform.

So on to my questions... Given that any module may have an XS module somewhere in it's dependency chain, do I have to keep a separate set of all the libraries for each platform (rather than only the XS ones)? I want to let CPAN.pm resolve and build all the depedencies, but as far as I know, I can't tell it not to build XS modules, so I can do those manually for each of the platforms.

There's also the problem that at some point a new platform may be added. When that happens, we need to build all the modules for that platform. Our current solution is to have a big make file which supposedly builds the right version of everything. In reality, this never works, and someone has to spend days manually building all the modules. Is there a good way to solve this?

Any other general advice on how to manage XS modules on multiple platforms?

Comment on Dealing with XS modules on multiple platforms
Re: Dealing with XS modules on multiple platforms
by moritz (Cardinal) on Jul 24, 2008 at 08:58 UTC
    It seems that your deployment process is tailored towards pure perl modules, and just not general enough.
    So on to my questions... Given that any module may have an XS module somewhere in it's dependency chain, do I have to keep a separate set of all the libraries for each platform (rather than only the XS ones)?

    I think so, yes. But it shouldn't be all that much work, because most pure perl modules are very easy to build once all dependencies are in place.

    But I'd chose a different approach: keep a local cpan mirror, and if some module makes trouble during build, write a patch and let your local mirror distribute the patched copy. Then just use CPAN to install everything on your machines.

    There's also the problem that at some point a new platform may be added. When that happens, we need to build all the modules for that platform. Our current solution is to have a big make file which supposedly builds the right version of everything. In reality, this never works, and someone has to spend days manually building all the modules. Is there a good way to solve this?

    That depends on why the build fails. If it's just management issues (for example dependencies built in the wrong order), you could try to set up a bundle that has all of the to-be-installed modules as dependencies, and let CPAN or CPANPLUS do the work for you. If the build fails because some module just doesn't work on that platform, I see no way around manual building.

    Disclaimer: I never maintained a larger cluster of machines, so take these thoughts as ideas only, not as advice.

      Thanks for this. I like the idea of running a local CPAN repository to take care of hacks (and versions). I have a feeling I can do this so that the local repository just goes to a real one if it can't find what it's been asked to build. We could potentially then use a Module::Install script to make it easier to allow for a re-build for a new platform.

      It still means I'd have to maintain a separate dir of built modules for each platform, but not sure there's an easy way around that.

        Also see CPAN::Mini and CPAN::Inject for creating/keeping the local mirror and the "distroprefs" feature of CPAN.pm for maintaining your local patches integrated in your mirror and automatically patching any module upon build time.

Re: Dealing with XS modules on multiple platforms
by Bloodnok (Vicar) on Jul 24, 2008 at 10:07 UTC
    IMO, I'd do something like...
    • have a perl server for each 'flavour' of supported OS - exporting the perl directory.
    • for any host, hard mount the perl directory from the appropriate server i.e. the perl server having the matching OS.
    Then it becomes merely a task of running make test install on each of the perl servers - a task easily accomplished with a suitable make(1) script on one of the servers that runs make using rsh(1) on each perl server.
    Moreover, the above would be simplified (almost immeasurably) if you have a central includeable (if there isn't such a word, there ought to be:-) make(1) script defining the expected/required build state in terms of modules.

    In that way, you will...

    • Only have to build one platform specific version of the libraries
    • Get a scaleable solution.
    • Have to re-run the build iff either a module changes (version) or a new module is introduced.

    HTH ,

    At last, a user level that overstates my experience :-))

      Hi. Thanks, but I'm not sure this really solves my problem. Deploying the modules is not an issue (I don't mind where they're deployed, the Perl code using them can just include the right path). I just want an easy way of building them for each platform with as little overhead and chance of making mistakes as possible.

      I'd also like to avoid building things as they're deployed. This can solve some of the issues, but it introduces another thing that can go wrong in the deployment process.

        Hi,

        If my suggestion is, at least in part, acceptable, the deployment is restricted to once per platform flavour ... c/w once per consumer/user.

        The suggestion regarding the common includeable make(1) script presumes that the same build standard is required on all platform flavours.

        If the standard make test install fails, then you've probably got bigger worries. Note also that, without intending to be offensive, it almost sounds like you aren't aware that...

        • the test target causes a build of the whole module - including any XS elements.
        • the install target is attempted iff the test succeeds.
        Thus, the build is simplified ... as is the deployment and as I say, if something like the topology I suggested is employed, the amount of either is reduced.

        Hope that clarifies things (a bit) ,

        At last, a user level that overstates my experience :-))

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://699808]
Approved by moritz
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (12)
As of 2014-12-19 15:57 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (85 votes), past polls