Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery

Yes, even you can use CPAN

by Tanktalus (Canon)
on Jun 24, 2008 at 21:31 UTC ( #693828=perlmeditation: print w/replies, xml ) Need Help??

Time and again, we see monks asking for solutions that "don't use CPAN modules" due to some artificial restriction placed on them. This restriction may be from their manager, from their IT department (or other development support team), from their web host, or even lack of knowledge on how to install CPAN modules without having root authority. This is how I've managed to work around all of these. I put it out here in the hopes that it helps other monks work around their restrictions.

No root authority?

The first step was learning how to install modules without root. Though I generally have root authority on my boxes, I found it informative and it actually has become the basis for some of my other work. The key has been to set PERL5LIB to include the path you're using to install to, e.g., $ENV{PERL5LIB} = "$ENV{HOME}/lib/perl5:$ENV{HOME}/lib" (I think some of the modules install differently when it's ~/lib instead of, say, ~/myperllib, so I found I had to add both) and then pass in the correct parameters to the .PL script the module comes with, e.g.: $^X -I$ENV{HOME}/lib Build.PL --install_base $ENV{HOME}/lib or $^X -I$ENV{HOME}/lib Makefile.PL LIB=$ENV{HOME}/lib PREFIX=$ENV{HOME}. The -I flag may be optional - but I like being explicit.

Once you have everything extracted and then the Makefile or Build files created, go ahead as normal, as if you had root. You just need to ensure PERL5LIB stays set, or that your scripts do a use lib ... properly before trying to use or require anything installed privately like this.

Web host doesn't want to install new modules

Unfortunately, many hosts don't find that they have a vested interest in your success. And, for smaller sites, this may be largely true. Bigger sites may be able to better command the attention of their host, so this probably doesn't apply to you (but then again, if you're a monk doing development for such a site, you probably don't need my help here).

In this case, I've taken everything I learned from non-root, and simply applied it here. If you have ssh or telnet access, this is trivial. If, like me, you don't, well, things can get trickier. In this case, I've taken the above algorithm, put it in code, and then uploaded it to my web host. Then I inserted a cron job to run this once a day to ensure that if I upload more CPAN modules, they get extracted and installed properly. When I add a new CPAN module, I simply adjust the cron job via its web interface to run two minutes from now, and then wait before uploading any of my other code that may rely on it. (In the meantime, I'm considering looking for a host that can compete with this one on price, but gives me ssh access, aka alternate solution #1.)

Manager support

If the problem you're having is at $work, and that problem happens to be the person who signs off on your paychecks, the problem becomes much bigger. It's no longer technical, which is unfortunate because monks are generally much better at technical solutions than social ones.

Here, I suggest starting simple. First off, and I need to stress this, I cannot recommend working around your manager. Do not do this. Your manager's responsibilities include managing resources, and you are a resource. Documenting your disagreement is fine, but don't try doing your manager's job for him/her.

Working with your manager instead of against them, find out what concerns your manager has about using CPAN modules, and see if you can address them without making it sound like you're attacking your manager for having these concerns. Remember: most CPAN modules' licenses are the same as Perl itself, so legal concerns should be there for perl as much as the modules.

Personally, I found this part really easy. When I explained to my manager how I'd be getting a bunch of working, tested code for free, he was all for it. But my manager was from a technical background (he had my job before me though it didn't involve perl until I got here). You may not be so lucky. If you aren't so lucky, just remember to deal with your manager based on his or her concerns, not yours. (This is also called "managing your manager" which is covered very extensively elsewhere.)

IT department reluctance

This is probably the most blatant problem. Though I use the term "IT department" here, it can apply to any similar scenario where the limitation is corporate, but not in your management chain. There are many ways to address this, and they largely re-use ideas from above. First off, you need your manager on your side. That is, you need to convince him/her that this is a problem worth solving. Try to do so without actually talking about how to solve it, but focusing on the why it needs to be solved aspect. This is because sometimes the problem looks intractable, and focusing on the mountain you want to cross may miss the road that passes through it.

Once your manager is okay with the solution to the problem, you will need to decide on how to solve it. Here there are both social and technical solutions, and the one you use may depend on the situation, or even the module you need installed.

The social solution is much like dealing with one's manager above. It involves talking to your IT department and determining the reason for the policy. My experience says that they're generally quite willing to share - though there are always people in any line of work that like to be arses, most policies come about from genuinely wanting to make the product better, even if we may think the implementation is misguided. See if you can address their concerns in a way that they will feel comfortable addressing your concerns.

In my case, when I wanted XML::Twig installed, the infrastructure team complained about a lack of resource. My solution was to trade some work: they had an assignment that was going to eat up large chunks of their time that not only made more sense for my team to do, but that I could automate easier and more reliably than they could anyway (thus I could gain better visibility, seem more productive, etc., so I wanted it anyway). So I suggested that I take on that work, if they'd simply install XML::Twig to make it easier for me to do. Though I had been trying to get XML::Twig installed for other reasons for years, this attempt took about 10 minutes, because I was addressing their concern: resource. Even though they had to fight a bit with expat on some platforms, overall they saved time, which meant they could do the other stuff on their plates faster (or at least sooner).

The technical solution I settled on for other CPAN modules that did not have external dependencies like expat was checking in tarballs from CPAN into our version control system, and then running the automated install tool (see the "web host" section above) to install all the tarballs from the CPAN directory into my lib directory. By adding this tool into the regular build's makefiles as a dependency that all my other perl programs depended, I could then count on using all my favourite modules whenever and wherever required.

I'm sure this doesn't cover every scenario - just the ones I've encountered and gotten past. Hopefully this is enough for newer monks to use to get past their roadblocks so that they can properly abuse, er use, the power that perl, with CPAN, gives them. Instead of re-inventing wheels, we can get on to inventing new vehicles that sit on those wheels.

Update (Mar13/2009): bellaire provided more things to think about: Top Seven (Bad) Reasons Not To Use Modules

Replies are listed 'Best First'.
Re: Yes, even you can use CPAN
by toolic (Bishop) on Jun 25, 2008 at 01:07 UTC
    ++ Tanktalus.

    I had been getting closer to posting something on this topic, in response to the somewhat common requests from Seekers of Perl Wisdom to provide help -- as long as it did not involve non-core modules. You did the work for me, providing comprehensive scenarios which exceed my experience level.

    Some simple advice I have given to fellow wisdom seekers is to navigate to the appropriate CPAN webpage, click on the "Source" link, and just copy'n'paste the code directly. This is only practical for simple, pure Perl modules, but there are plenty of those available.

      For pure-perl modules, use Inline::Module, and it will automatically copy/paste the module into your program
Re: Yes, even you can use CPAN
by zby (Vicar) on Jun 25, 2008 at 08:58 UTC
    What I've discovered recently is that if you want to be completely independent or if you need to use a newer version of Perl than is what installed system-wide then it is also quite easy to install your own Perl interpreter in a private directory (just remember to follow INSTALL and add -Dprefix='/some/dir' to Configure's args). Then you just add this new perl into your PATH - and you can use perl -MCPAN -e shell or cpanp as usual without fiddling with parameters to the *.PL scripts.
Re: Yes, even you can use CPAN
by holli (Monsignor) on Jun 27, 2008 at 07:00 UTC
Re: Yes, even you can use CPAN
by monarch (Priest) on Jun 25, 2008 at 20:10 UTC

    Nice write-up, Tanktalus. Interesting trick for building modules on a remote web host.

    I've tended, instead, to untar all the modules that I need locally, and just keep the pure Perl versions and FTP those up, ensuring that they can be found in my script with an appropriate use lib... statement. This was how I managed to get TWiki working at the web host provider iPowerWeb (although I've since moved to my own virtual linux hosting).

Re: Yes, even you can use CPAN
by rafl (Friar) on Jul 09, 2008 at 12:26 UTC
Re: Yes, even you can use CPAN
by Anonymous Monk on Jul 16, 2008 at 04:04 UTC
    One of the reasons I've been reluctant to use modules in my code in the past has also been accessibility to CPAN.

    Typically my environments have been shut away from the outside world of the Internets, and it can take hours to follow and trace all of the pre-reqs for a given module by hand, downloading to my local machine, then uploading to the environment, only to find out that I missed one or two... wash, rinse, repeat for a few hours just to get one module finally installed.

    How did I get over this? I carved out 6 gig of disk on a linux box in my environment and created a CPAN mirror. My friendly network admins open up the firewall for me when I need to update via rsync occasionally and I just point all my local boxes to that server internally. After my updates they close it back down. This way I can install modules to my hearts content as I need them.

    Of note, I have seen software produces where an entire distribution of Perl has been included (HPOM/Dazel comes to mind) I imagine to get over the issues the above poster has noted....
Not everywhere...
by songmaster (Sexton) on Jun 27, 2008 at 03:02 UTC

    My primary work product is a fairly large open source software system with users around the world. Perl is currently only used for configuring and building the other software, which is mostly written in C and C++. I am rewriting some of its Lex/Yacc/C-based programs in Perl for increased flexibility, but we don't want to have to support all our external users (who don't pay our salaries) in installing modules from CPAN and handling any version dependencies that my code has.

    We can (and do) say "this software requires Gnu Make 3.81 and Perl 5.6 or later" since they are not a hurdle for most users (other than those poor guys forced to work on Windows), but to add "and the CPAN module Acme::Blah 2.2" would leave new users puzzled, increase the traffic on our mailing list, and might scare some of our users away completely.

    Luckily our needs are light and most things we need are available in the core distribution so it's not that much of a loss. This is therefor one scenario where CPAN isn't necessarily useful (sorry to burst the bubble).

      You don't need to leave users puzzled. You can simply put the CPAN modules in with your build environment, and have make files install them. That's what I do at $work (see IT department reluctance in my original node). Our make files run a script that goes and untars the modules and runs the install process. If they're pure-perl modules, you may even be able to get away with pre-installing them (i.e., just placing their files in your tree). Of course, since you're then distributing the result, you'll want to consult with the lawyers about proper attribution and licensing, but that should cost a fraction of what developing some of the modules from scratch would be anyway.

      The only thing I can see where you can't just install or upgrade willy-nilly might be perl itself, or other external dependencies (such as Oracle if you want to use DBD::Oracle, or expat for XML::Parser). Most modules, however, should not have such a limitation. With a little thought, most other issues should be easily worked around, and even external dependencies can sometimes be worked around, too.

      How does this prevent you from bundling these modules with your software? Or having your installer say something to the effect of:

      Oops, you don't seem to have the Acme::Blah Perl library installed. This is required for proper operation. Should I download it for you?
      (sorry to burst the bubble)
      Your choice of words can work for or against you :)
      the act of puncturing with a small point; "he gave the balloon a small prick"
Re: Yes, even you can use CPAN
by Skeeve (Vicar) on Apr 09, 2015 at 06:44 UTC

    Yesterday I wanted to install Config::General on a system at work. But the admins in their wisdom removed (or did not install) ExtUtils::MakeMaker. When I looked at MakeMaker's source I was a bit astonished to see use ExtUtils::MakeMaker in it, but nevertheless gave it a try and it worked.

    So for future reference (of myself and others) here is what I did, after downloading the source and unpacking it in my home directory to get my modules up:

    cd ExtUtils-MakeMaker-7.04/ perl Makefile.PL INSTALL_BASE=/opt/sw/perl make make install cd ../Config-General-2.56 export PERL5LIB=/opt/sw/perl/lib/perl5/ perl Makefile.PL INSTALL_BASE=/opt/sw/perl make make install perl -MConfig::General -e 42

    The last step was to verify it worked.

Re: Yes, even you can use CPAN
by OfficeLinebacker (Chaplain) on Apr 20, 2013 at 16:58 UTC
    Question: when you say

    $ENV{PERL5LIB} = "$ENV{HOME}/lib/perl5:$ENV{HOME}/lib"

    Don't you want to actually apppend the libs desired to the variable, rather than replace the variable's contents with the directory you're installing into?

      Don't you want to actually apppend the libs desired to the variable, rather than replace the variable's contents with the directory you're installing into?

      it doesn't matter / it depends on your setup

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://693828]
Approved by kyle
Front-paged by planetscape
[Corion]: On the upside, I spend a lot of time thinking this weekend about how to actually implement rate limiting for futures, and if things work out, maybe even loading a configuration from an external file makes sense
[Corion]: I've also found some interesting invariants that I have to think/write about more. A simple rate limiter will never change the order of the input, while a limiter that allows for parallel execution will change the order. But my API currently allows for bo
[Corion]: ... for both, and I'm not sure if I want to add the cruft from the parallel API (a token that you need to hold on to while you hold the lock) to the rate limiting API too, to allow seamless up/downgrades, or not.
[Corion]: Also, rate limiting will look great with await: my $token = await $limiter-> limit($hostname); instead of my $f = $limiter->limit( $hostname )->then(sub { my( $token)=@_; ... });

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (9)
As of 2017-10-23 08:25 GMT
Find Nodes?
    Voting Booth?
    My fridge is mostly full of:

    Results (277 votes). Check out past polls.