Automating CPAN Configuration

[dsheroh]

I really didn't want to have to ask this question here, as it seems like something that should already have a well-known solution, but neither google nor super search was able to find such a solution for me. So...

I need to distribute a few CPAN modules to approximately 100 hosts, some of which already have CPAN configured, but most do not. According to the CPAN documentation at http://search.cpan.org/~andk/CPAN-1.87/lib/CPAN.pm:

How do I install a module and all its dependencies from the commandline, without being prompted for anything, despite my CPAN configuration (or lack thereof)?

CPAN uses ExtUtils::MakeMaker's prompt() function to ask its questions, so if you set the PERL_MM_USE_DEFAULT environment variable, you shouldn't be asked any questions at all (assuming the modules you are installing are nice about obeying that variable as well):

% PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install My::Module'

Unfortunately, this is not reliably the case, as several of the hosts I've tested this against have gotten themselves stuck in an endless loop due to "Select your continent (or several nearby continents)" defaulting to an empty response, which is not valid. (So much for the claim that it will work "despite my CPAN configuration (or lack thereof)"...)

My next idea was to copy CPAN/Config.pm to each host and just duplicate a single good configuration on all of them, but that sits inside a version-specific subdirectory and the hosts in question are not all running the same version of perl, so that doesn't appear to be a strong candidate for automation either.

What other options do I have for getting all of these hosts configured for CPAN without having to touch each one manually to tell it "no, I don't want to do manual configuration, so just go ahead and pick some default values instead"?

[wsheehan]

Have you considered using expect? Here is a sample expect script that will spawn the CPAN shell, answer "no" if appropriate during the configuration prompt, and quit when it sees the normal cpan> prompt. If the manual config prompt doesn't appear, the script will still catch it. This was written in the previous few minutes, so it can really be cleaned up and set to do something other than smartass remarks when something goes wrong; however it seems to do the job:

# 10 second time limit for one of our conditions to be met
set timeout 10

eval spawn "/usr/bin/perl" "-MCPAN -e shell"

expect {
    "Are you ready for manual configuration?" {
        send "no\r"
    }
    "cpan>" {
        send "quit\r"
        expect eof
        exit
    }
    timeout {
        puts "\nWe ran out of time\n"
        exit
    }
    eof {
        puts "\nOh noes, unexpected EOF\n"
        exit
    }
}

# This will only be reached if a default configuration keypress was re
+quired
expect "cpan>"
send "quit\r"

[dsheroh]

I hadn't thought of using the expect command itself, although Expect.pm did come to mind a couple hours after posting my question. I suppose that's the most sensible way of handling this, given that the code to actually do the remote module installs is Expect.pm-based, but I was still hoping to find something more streamlined even after I thought of it. (I'm... less than fond of Expect, as it seems to enforce a style of coding that feels ugly and cumbersome to me. But I can't think of any model that suits my taste better without giving up a lot of the flexibility and power of expect, so I guess I can't complain all that much about it.)

So I guess I'm off to roll this into the existing code. Thanks for the suggestions, everyone!

[Withigo]

Once upon a time I had to do almost the exact same. What I ended up doing is by no means elegant, but it did work for a couple dozen or so separate hosts, all of which could have been running different os'es and perl versions. Hundreds of modules were automatically installed on each host without any problems. (Excepting modules which themselves required manual input to install, but they all had environmental variables which could be provided to automate any prompts).

# get the location of CPAN::FirstTime.pm
export PREFIX=`perl -V:privlib`
${PREFIX##"privlib="}

# force CPAN::FirstTime to not default to manual
# setup, since initial CPAN setup needs to be automated
perl -pi -e'$. == 73 and s/yes/no/' $PREFIX/CPAN/FirstTime.pm

# make CPAN set itself up with defaults and no intervention
perl -MCPAN -MCPAN::Config -MCPAN::FirstTime -e'CPAN::FirstTime::init'

# undo the change
perl -pi -e'$. == 73 and s/no/yes/' $PREFIX/CPAN/FirstTime.pm

Now that CPAN is prepared for use the `PERL_MM_USE_DEFAULT=1 perl` etc... should work. I also distributed a pre-written Config.pm to all hosts; but you say you'd like to avoid doing that, so this may not work perfectly without also providing your own Config.pm. I do remember that my Config.pm only added a couple values, as almost all of the default values worked just fine. Unfortunately I've forgotten which values I needed to add, but a few minutes of trial and error would reveal them. The previous comment about setting $CPAN::Config->{urllist} would be one way of setting those values. Good luck!

[dsheroh]

I'm not really against distributing a pre-configured Config.pm to each host, it just appears that, to do so, I would need to check the remote perl version to determine where to put it and, once I'm to the point of calling perl -v, parsing its output, building the right path, and putting Config.pm there, then it would be simpler and more reliable to just fire up the CPAN shell, say "no" if it asks about manual config, and then exit from it.

[Anonymous Monk]

Right, the mirror selection is the one thing that doesn't have a default. There's a trick to this.

require CPAN::Config;

$CPAN::Config->{urllist} = [ $mirror_url ];

require CPAN;

CPAN::install('My::Module');

If memory serves, that should work fine - or if not something pretty close

- Matt S Trout

[Anonymous Monk]

Don't :)

1) use a package manager instead of -MCPAN. IE, if you're on a debian system and you want to install DBI and the mysql drivers, do

 apt-get -y install libdbd-mysql-perl libdbi-perl

Easy to automate across your 100 machines.

This assumes someone (perhaps yourself) has already created packages of these modules.

2) create a directory structure for your perl modules. (IE /organization/opt/perl-modules/.) On a master machine (build box), install your modules into this hierarchy. Then rsync the 'perl-modules' directory structure to the 100 boxes.

CPAN as an archive is awesome. -MCPAN as a method for software installation .. not so much. It was cool when I discovered it back in the day, but now I really really really wish that CPAN defined a standard 'perl module package format' -- a single archive file containing the module, and a package manager tool to add/list/remove packages. Failing that, it would be cute if CPAN would automagically create .debs, .rpms, or Solaris pkgs.

[adamk]

Agreed on the package manager thing. _If_ your package manager supports all the modules you need, with all the versions you need, then you should be using that. Unfortunately, the number of CPAN modules is higher than the total package count for several distributions, so that isn't always (often?) possible.

[dsheroh]

Don't

Don't use CPAN? What heresy is this? Where's the real monastery and how did this impostor manage to take its place?

Personally, at home, I do prefer to use Debian's package repository to install modules rather than going to CPAN for them. In the current case, though, I'm working with a client whose change management policy makes it cumbersome to install packages from the OS vendor, but CPAN flies under its radar, so that's the admin team's preference. Also, the current *nix environment is a mix of Red Hat and HP-UX, with an influx of Solaris looking likely in the mid-term future; although RH seems likely to have several modules packaged, I'm not so sure that I'd expect the same from HP and Sun. This mixed environment would also obviously complicate any attempt at setting up a (set of) master build box(es) unless it's restricted to only pure-Perl modules.

CPAN may not be the best solution (doing a separate make on each host is inefficient, if nothing else), but it works well enough.

[doc_faustroll]

I was doing some automation and auto bootstrapping of CPAN, but since I was creating configure and enviro bootstrap scripts to set up an environment for a Java application on n hosts, I elected to use PAR to bundle the scripts into executables that can be deployed to hosts fully formed with modules installed and a standardized version of Perl included, if need be.

What in particular is the application or script or suite of scripts that needs these modules installed? Is automation of CPAN install really desirable when you don't know the version of Perl on these hosts? In the case of that kind of heterogeneity, I would go with PAR. Unless, of course there is some compelling reason to work with CPAN auto install.

[dsheroh]

What in particular is the application or script or suite of scripts that needs these modules installed?

I'm currently supposed to be a hired gun helping out this company's sysadmin team, but they quickly noticed that I have a strong Perlish bent and decided they'd rather have me making tools to help the admins out instead of actually doing admin work. (Spend my time writing code instead of answering calls from users? Sounds good to me!)

The particular apps/scripts/suite, then, is the few bits of what I've been doing which need to be run locally on each host rather than from a central location. So not a lot there that's likely to be dependent on specific versions of either perl or any modules. I suppose there's the possibility that a module could be incompatible with a certain version of perl, but none of the versions in use here are that old - it's mostly 5.6.1 with a bit of 5.8.0 and 5.8.5 mixed in here and there.

I'd be interested to hear more about PAR, though - I've never run across it before. Can you recommend any links?

[doc_faustroll]

the pp packager from the PAR distribution works quite well, although you must thoroughly test the packaged executable on a sample of target machines. I have noticed that an occasional dependency is not picked up.

A recent client was extremely frustrated when an executable did not work on one of their target machines even though it passed all tests on mock target machines. I added an additional runtime check -c to the pp call and that fixed the problem.

you can install PAR via cpan. pp will be a part of the install.

I myself would like to see a list of issues or problems that people have had with pp. I'm using it in a similar admin centric tool dev way that I anticipate you could be using it and it is working well for me. You should explicitly declare your use of modules at the top of your script, especially if you are using dbd modules rather than just load them within DBI.