Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid

Distribute script with dependencies

by kcalvelli (Novice)
on Oct 25, 2013 at 12:30 UTC ( #1059639=perlquestion: print w/replies, xml ) Need Help??
kcalvelli has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I wrote a script that uses a number of CPAN modules that my company would like to deploy on a variety of locked-down servers (AIX, Linux - no internet access). I was using pp to create an executable, but they do not want to distribute a binary. The assumption is that there will be a working perl on all servers, so I just need to bundle the script with all dependencies.

The ideal solution for them would be to keep the script with all dependencies in a single directory under version control (we use svn). I looked into using local::lib to do this, but it didn't seem like a good fit (or I just couldn't figure out how to make it a good fit). They would like to be able to simply zip up one directory, drop it on a server (again, either Linux or AIX), unzip it and be able to run the script.

Is there a common solution for this problem? I have been looking for a couple of days and haven't come up with anything that seems to be a perfect fit.

Any help would be appreciated! Please let me know if I need to provide any additional information.

Replies are listed 'Best First'.
Re: Distribute script with dependencies
by Limbic~Region (Chancellor) on Oct 25, 2013 at 12:47 UTC
    Having a working perl on all servers is not sufficient. Are they all the same version of Perl compiled with the same options? Are there any modules that rely on system libraries that might not be installed such as Math::Pari? The point I am trying to make is that the "they" and "them" you are referring to are making your job a maintenance nightmare.

    With that said, I believe local::lib and a reliable way to change directories to the location of the perl script should be a viable solution in many circumstances (needs to happen in a BEGIN block prior to the use of the local library.

    Cheers - L~R

      Unfortunately, they are not all the same version of perl. I have been using a perl compiled with gcc on both Linux and AIX for development, but the new requirement is to use the system perl for AIX and Linux. I thought that might be a problem, which is why I have been trying so hard to push the pp solution (I have compiled different versions for both platforms and it works great).

      I have been asked to "remove any dependencies" from this particular script if I can't come up with a viable solution for distribution. I'd rather not re-write the wheel.

      Does anyone know of a good write-up on using local::lib for distribution?

        I might be able to make this easy for you. The system AIX Perl is built with the non-free propietary IBM compiler. As a result, you can not use any non pure Perl modules because they will not be binary compatible unless they have an expensive license. Update: Let me give a concrete example. Let's say they want to connect to a database (DB2, Oracle, Pg, etc). You can't build DBD::DB2 with gcc and have it work with the system Perl.

        Cheers - L~R

Re: Distribute script with dependencies
by Corion (Pope) on Oct 25, 2013 at 13:12 UTC

    If your script and its (non-core) dependencies are all pure Perl, you can get away by using App::fatpacker, which creates one huge script out of your modules and your script.

      This was the solution I went with.

      I had considered using App::fatpacker earlier, but discounted this option due to the fact that one of my dependencies used autoload. After weighing all options, I decided to hack up the dependency to remove autoload. Since everything was pure perl, I now have a big, fat script that "has no dependencies" as requested. So far we are running fine on RHEL6 and AIX7.1. Haven't tried on an AIX6 box yet ....

      Of course maintenance will be more challenging going forward, but no more challenging than many of my other options.

      Thanks, Monks!

Re: Distribute script with dependencies
by Tanktalus (Canon) on Oct 25, 2013 at 14:03 UTC

    What's the difference between distributing a binary and a script?

    What I'm planning on doing, with AIX no less, is to have version control contain the source to perl and 160+ modules, plus another repository with my actual code in it. We build the code, tar it up, and distribute it. I do not want to use the system perl because that creates headaches. First off, it's too old - AIX 7 comes with perl 5.10.1. Second, it can change outside of my control: an AIX update could add or remove stuff, including introducing a fix I'm not compatible with.

    In my case, we already have the "non-free proprietary IBM compiler" so that isn't the issue. But it's far better to ship your own copy of perl with your code. Then you don't need to worry about the OS compiler, or anything else. It also makes updating AIX easier - going from AIX 6 to AIX 7 broke some of our stuff and we had to build stuff separately for the two because we were building for the system perl (5.8.8 vs 5.10.1).

      ...going from AIX 6 to AIX 7 broke some of our stuff...

      Hijacking the OP's thread for a moment, can you describe what the problems were and what you did to resolve them? I would like to head them off if possible for my environment.

      Cheers - L~R

        Fair enough - the breakage was something that will likely strike you as obvious once you see it: going from 5.8.8 to 5.10.1 - that's not a binary-compatible version change. As I seem to recall, this was one of the biggest binary breakages p5p has ever done, but size doesn't really matter - any breakage is a breakage. Obviously, pure-perl code wasn't impacted, other than the change from 5.8 to 5.10 itself (yay, can now use state variables! -- but I didn't encounter anything significant here), it's the XS code, such as DBD::DB2, as you pointed out. We needed DBI, DBD::DB2, JSON::XS, and various other XS modules, so these got hit. Originally, TPTB wanted to use the same AIX 6 box for compiling both versions of our product, but this forced their hand - we had to get an AIX 7 system to compile.

        In addition, we were installing the CPAN modules to the site directory, which, again, changes from 5.8.8 to 5.10.1. If we installed the 5.8.8-compiled XS modules to the 5.8.8 directory on AIX 7, perl 5.10.1 on AIX 7 simply won't see them. If you're installing to a local lib directory, that's not an issue.

        So, basically, we had to compile on both AIX 6 (against 5.8.8) and AIX 7 (against 5.10.1) to get our modules working in both places. Going with our own private perl in the future, we could compile the whole stack on an AIX 6 box and use it on both AIX 6 and AIX 7.

        So, really, there was nothing here that was unique to AIX per se. Going from RHEL 5 to RHEL 6, for example, could have basically the same issues. It's pretty much all just the fact that we're installing CPAN modules to the system perl pre-compiled and automatically, without any compiler being on the target system.

Re: Distribute script with dependencies
by jellisii2 (Hermit) on Oct 25, 2013 at 12:54 UTC
    I would think Par::Packer would be the best bet here, despite the fact that it'll be a bigger file to cart around.

    The -p switch for pp will create the archive, but there's no clean way to unpack it on the target.

    Are all the targets within your control? Perhaps a darkPan option would work?

    Edit: Re^3: Distribute script with dependencies suggests that this won't work and that a packed executable is the best option.

Re: Distribute script with dependencies
by marto (Bishop) on Oct 25, 2013 at 12:58 UTC

    How about you create a PAR file with just your modules, code and data and distribute that to the target machines, rather than the stand alone executable route? See also the slides I link to here for info on how it works (pretty much like your 'zip up one directory' idea) and also the section on Application repositories. You may be interested in the mechanics of how it can distribute your app (and updates to it etc).

    Update: ah, this complicates matters.

Re: Distribute script with dependencies
by martell (Friar) on Oct 27, 2013 at 08:50 UTC

    Hi all,

    Pure out of curiosity and slightly off topic: has someone created a debian or rpm packages for distributing Perl programs? And what are your experiences with it?

    In the program I'm developing, I use a mix modules coming form the Debian package repositories and modules that I installed myself via CPAN because there wasn't a Debian package available. My use case is a bit different than the original one: I'm targeting computers that are connected via internet and are one flavour of Linux.

    I would like to create a debian package that contains all the necessary scripts and additional cpan modules I need. It would allow my to easily update my scripts and roll-out new features.

    I didn't research the topic yet, but in light of this question I would like to know if anyone has experience with it. And if you can give me some pointers to good relevant reading material, it would be a great help.

    Kind regards


        Thanks! Exactly what I was looking for!

        Kind regards


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1059639]
Approved by marto
Front-paged by ww
[Corion]: ... those in an automated fashion seems somewhat hard to me. Although it should do wonders for the test coverage ;)
[haukex]: Corion: I once wrote an automated thingy for that here
[haukex]: here's the code that uses it
[Corion]: haukex: Hmm - maybe I can reuse that. I see that it uses Pod::Parser, which I think was one of the more fragile parsers. But if I'm statically (re)generating the tests instead of doing that "live" on the client/tester machines, that's a much smaller...
[Corion]: ... problem space than trying to cater to all versions of Pod::Parser(s)
[haukex]: I think you're right, I think Pod::Simple is the preferred parser now
[haukex]: But I was just using it as an author test anyway

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (10)
As of 2017-02-27 12:22 GMT
Find Nodes?
    Voting Booth?
    Before electricity was invented, what was the Electric Eel called?

    Results (385 votes). Check out past polls.