How to clean-up an autobundle so it's really "auto-installable"

[sundialsvc4]

The prevailing wisdom, discussed recently here, is that you can simply use the autobundle command to snapshot your present installation, then install Bundle::whatever in order to replicate your installation.   But my experience is that ... this is neither true, nor efficient.

(1) The autobundle may include components that are built-in to the Perl distribution.   An attempt to install them, which is carried-out by the process of installing the bundle, will fail ... and, when it does, the entire autobundle-install will grind to a halt.

(2) The ordering of the modules in the autobundle is simply alphabetic.   Thus, modules like YAML, upon which so many other modules depend, will not be installed until nearly dead-last, and every install preceding it will not perceive that it is there, because indeed it is not there yet.   If the modules build themselves in different ways based upon what is and is not available, as many of them do, it becomes necessary to force(!!!) install the bundle, at least twice if not more.

Edit:   (3) If a newer version of Perl is being installed alongside an older one, all of the CPAN libraries which rely upon XS code must be reinstalled into the local directory, even if other libraries already exist.   (These would probably be compatible only with the older Perl, and its various co-dependencies, etc.)   In this case, it is even more important that modules be compiled in the proper order and that the “older Perl” @INC directories not be searched.

This is, shall we say, “conduct not becoming a fine language of good breeding,” which, as we all know, Perl is.

What I am looking for, therefore, is a tool that will “sanitize” an autobundle, both to remove built-ins from the list and to optimize the order of the list so that dependencies are installed early.   The process of re-creating a CPAN, e.g. after having upgraded Perl, ought to be a very clean and dependable procedure.   Surely, it is.   What am I missing?

[moritz]

It would seem that instead of creating a "bundle" (which seems to be a rather magic thing), it might be easier to just create an empty module that depends on the modules it should bundle.

Modern CPAN installers like App::cpanminus are quite capable of actually installing recursive dependencies, so the first installing attempt should succeed.

<update> It seems that what I suggested is implemented as Task, so maybe an autotask script would be in order? </update>

Granted, it neither answers your question (I don't have a tool that does it automatically) nor does it solve the problem of modules being compiled conditionally, but maybe it gives you food for though.

What you seem to want is a topological sort of the depency graph - maybe Sort::Topological and the data from http://deps.cpantesters.org/ could help you here.

[sundialsvc4]

Very probably so.   Specific details are needed with regard to your various suggestions.   I see no references in CPAN, etc. to tell me, “what is autotask?”

[moritz]

"autotask" is my suggested replacement (but not yet written by anybody) for autobundle, that works the same but creates a Task instead of a Bundle.

[sundialsvc4]

Well, they say that good Perl programmers are lazy.   Goody for the good ones.   Fortunately, I guess that programmers like me must be lazy, too.  

#!/usr/bin/env perl

use strict;
use warnings;

use Data::Dumper;

print STDERR "Scanning ...\n";

# find dependent things ... note 'scandeps' only considers modules.
# note: do not say "*.pm" here or you won't look in subdirs

# replace 'path_to_my_app_libs' with whatever is correct for you ...

my @output  = qx{scandeps.pl /path_to_my_app_libs/*};  

my @keepers;

# The output of interest is the module-name, found in single quotes

foreach (@output) {
    chomp;
    next unless /^'([A-Z].*?)'/;
    push @keepers, $1;
}

print STDERR "Installing ...\n";

foreach (@keepers) {

    # The output is a little "wordy."  
    # Omit hundreds o' things that we know are really parts of others.

    next if /^Date::Manip::TZ::/; 
    next if /^Date::Manip::Offset::/;

    qx{cpanm $_};                   # install it
}

# see if we got everything ...
# (failure of some of these is not necessarily awful)

foreach (@keepers) {
    print STDERR "require $_\n";
    eval "require $_";        # magic necessary voodoo ... see 'perldo
+c require'
}
print "Normal completion.\n";

At first I was seriously flummoxed by the fact that I did not get complete results.   It turns out that it is very important that the scandeps.pl command ends with “*” and not, as originally written, “*.pm”   (The first form omits subdirectories.)

This script requires App::cpanminus (for cpanm) and Module::ScanDeps (for scandeps.pl).

As written, it still does not make any improvements to the order in which things are attempted.   I took a brief look at the hashref returned by scan_deps() and it does look like it could be used more fully, i.e. in order to do that.   (No doubt, it already has...)

[Anonymous Monk]

Here is my version, takes bundle (or as cpanp names it, snapshot) file as argument

#!/usr/bin/perl --
use strict;
use warnings;
use CPANPLUS::Internals::Utils ;
use DBI;
Main( @ARGV );
exit( 0 );
sub Main {
    my @snaps = @_;
    my %U;
    my $db = CPANPLUS::Internals::Utils->_home_dir . '\.cpanplus\db.sq
+l'; 
    
warn "db is $db
snaps is @snaps
";
    
    my $dbh = DBI->connect(
        qq!dbi:SQLite:dbname=$db!,
        undef, undef, { RaiseError => 1, PrintError => 1, },
    );
    my $sth = $dbh->prepare( q! select path, package from module where
+ module = ? ! );

    for my $file ( @snaps ){
        open my($fh), '<', $file or die;
        while(<$fh>){
            if(/=head1 CONTENTS/.. /=head1 CONFIGURATION/){
                if( /^(\w\S+)\s/ ){
                    my $module = $1;
                    $sth->execute($module );
                    my($a,$p) = $sth->fetchrow_array;
                    next unless $p and $a;
                    $a =~ s!^authors/id/!!;
                    next if $p =~ /^perl-5\./; 
                    next if $p =~ /^perl-5\./; 
                    my $pp = $p;
                    $pp =~ s/-\d.\d.+$//; ## libwwwperl
                    $U{$pp}="$a/$p";
#~ perl-5.10.1.tar.gz
#~ perl-5.11.1.tar.bz2
#~ perl-5.12.3.tar.bz2
#~ perl-5.13.9.tar.gz
                }
            }
        }
        close $fh;
    }
    undef $sth;
    undef $dbh;
    
    
    print "\n\nsystem qw[ cpanp -d \n";
    print "      $U{$_}\n" for sort keys %U;
    print "\n\n];\n";
}

__END__


system qw[ cpanp -d 
...
      G/GB/GBARR/libnet-1.22.tar.gz
      G/GA/GAAS/libwww-perl-6.02.tar.gz
];

I use cpanp -d to first download all the tarballs then later, offline, change -d to -i to install all the tarballs

[Anonymous Monk]

Sometimes the simple solution isn't pure Perl.

cpanm `cat Snapshot_2014_12_29_00.pm|awk '/\:\:/{print $1}'`

I suppose I could find a way to do that in Perl but awk is simple and fast. This certainly puts lots of extra modules on cpanm's list, but it knows how to figure this stuff out.

[Anonymous Monk]

Sometimes the simple solution isn't pure Perl.

cpanm `cat Snapshot_2014_12_29_00.pm|awk '/\:\:/{print $1}'`

I suppose I could find a way to do that in Perl but awk is simple and fast. This certainly puts lots of extra modules on cpanm's list, but it knows how to figure this stuff out.