Re: ExtUtils::Installed doesn't list all modules

[Intrepid]

rinceWind wrote (originally at 13:51 GMT +5 on 24 Sept 06):

Sadly, I find that EU::I is missing huge swathes of modules on my system. Diving into the code, I find that it only looks in two places: $config{archlibexp} and $config{sitearchexp}. Why oh why, I ask myself, just these special places? Surely the code should be traversing @INC. Heck, you could even have custom directories added via use lib or $ENV{PERL5LIB} that you want to be visible.

I've just looked at the code for ExtUtils::Installed and I have some answers. Not the asked-for solutions, but some answers. It's become de rigeur to bash ExtUtils:: generally (?). My opinion is: let's at least be correct when we do.

Why, I ask myself, just these special places? Surely the code should be traversing @INC. The code is traversing @INC in its search for the module names found via the mechanism I will describe shortly. See line 116 in the ExtUtils/Installed.pm shipped with perl-5.8.8 by ActiveState, for example (that's 0.08). Or refer to the code to follow my explanation if you don't have an editor or other Web tab open on the source already. See the new() subroutine specifically.

The new sub defines two vars as below:

      my $archlib = $Config{archlibexp};
      my $sitearch = $Config{sitearchexp};

and IMHO these two vars need to be joined by a third var, $Config{vendorarchexp}. That's a flaw in the module under almost any reasonable scope of expectations that I can conjure in my mind. However, it is not unreasonable that these two existing Config vars are used as they are, and this use is not limiting the scope of the search in exactly the manner explained by the OP. The reason for examining only FOOarchexp is that the values associated with each are the locations under which .packlist files are found.

The .packlist files (familiar to those of us who watch the actions of module installers (ExtUtils::MM, Module::Build) closely) were installed when each module was added to the Perl installation. ExtUtils::Installed is using the presence of these .packlist files as a means to the end of listing what modules have been installed.

Edit 25.09.06: correct next statement

It does not read the .packlist file in order to locate the module, you'll notice (because you are really reading the code by now, right? -- because it's more interesting than this long-winded explanation); ... the code simply takes note of the path to each (under .../auto/ ) and then creates a module name from that pathname. This is pretty reasonable behavior actually (in my analysis) in that it correlates well with how Perl's long-established mechanisms for how module management works.

[rinceWind]

Thanks for the insights. I have been digging a bit further.

... and IMHO these two vars need to be joined by a third var, $Config{vendorarchexp}. That's a flaw in the module under almost any reasonable scope of expectations that I can conjure in my mind. However, it is not unreasonable that these two existing Config vars are used as they are, and this use is not limiting the scope of the search in exactly the manner explained by the OP. The reason for examing only FOOarchexp is that the values associated with each are the locations under which .packlist files are found.

The .packlist files (familiar to those of us who watch the actions of module installers (ExtUtils::MM, Module::Build) closely) were installed when each module was added to the Perl installation. ExtUtils::Installed is using the presence of these .packlist files as a means to the end of listing what modules have been installed.

Firstly, I don't see a config parameter called vendorarchexp. I am using the perl packaged with Debian. Maybe vendorarchexp is only set when perl has been built with vendor specifics. Here's the tail of my Config.pm:

tie %Config, 'Config', {
    archlibexp => '/usr/lib/perl/5.8',
    archname => 'i486-linux-gnu-thread-multi',
    cc => 'cc',
    d_readlink => 'define',
    d_symlink => 'define',
    dlsrc => 'dl_dlopen.xs',
    dont_use_nlink => undef,
    exe_ext => '',
    inc_version_list => '5.8.7 5.8.6 5.8.4 5.8.3 5.8.2 5.8.1 5.8.0',
    intsize => '4',
    ldlibpthname => 'LD_LIBRARY_PATH',
    libpth => '/usr/local/lib /lib /usr/lib',
    osname => 'linux',
    osvers => '2.6.15-1-686',
    path_sep => ':',
    privlibexp => '/usr/share/perl/5.8',
    scriptdir => '/usr/bin',
    sitearchexp => '/usr/local/lib/perl/5.8.8',
    sitelibexp => '/usr/local/share/perl/5.8.8',
    useithreads => 'define',
    usevendorprefix => 'define',
    version => '5.8.8',
};

Secondly, I can think of several use cases where .packlist files are living elsewhere. The one that is biting me here is that I am on a system which has had its Perl version upgraded. The machine was originally Debian Sarge, which comes with 5.8.4, and is now Etch, with 5.8.8. Here's my @INC:

    /etc/perl
    /usr/local/lib/perl/5.8.8
    /usr/local/share/perl/5.8.8
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.8
    /usr/share/perl/5.8
    /usr/local/lib/site_perl
    /usr/local/lib/perl/5.8.4
    /usr/local/share/perl/5.8.4

And I find that my .packlists are evenly divided between being under /usr/local/lib/perl/5.8.4 and /usr/local/lib/perl/5.8.8. I don't believe I have a broken perl installation.

Consider the user installing modules not as root. They don't have write permissions to any of the perl lib directories, so they set PERL5LIB and install to their own private directory tree, packlists and all. This is the scenario for me at work; I am a developer, not a sysadmin.

Finally, it occurs to me that the packlists might not be giving the whole picture, even if we are picking up all of them. We know that current versions of ExtUtils::MakeMaker and Module::Build respect and honour packlists, but Module::Build has only done this relatively recently. I've also found cases of Module::Build installs where the module author has hijacked the install mechanism, and there's no packlist in sight. And, what's to say about other package management schemes. ActiveState PPM probably respects packlists, but I believe it's up to the module packager for Debian dpkg, Redhat RPM, yum, etc. to include the .packlist (or not, as might be the case).

[Intrepid]

I'm preparing my swing (a module which will be my effort to address this problem you've raised. I'm just posting this now as a message and a 'placeholder' to make myself obligated to posting my effort ;-). This node will be updated later.

Update: 25.09.06: added all below.

OK. While not perfect (and I am a wee bit hurried in making this posting even), there's a solution to follow, YMMV, etc etc. We don't want to locally modify ExtUtils::Installed for reasons rinceWind so cluefully mentioned (a future upgrade could well just clobber our modifications someday). So let's create a new module and stick it in the sitelib where (hopefully) no harm will come to it.

Since ExtUtils::Installed is doin' the OO mambo, let's dance with it. We'll just make our own constructor (which ordinarily is called just once per run). See how this works for you, rinceWind.

Update: 26.09.06: CODE CHANGE -- found a regexp mistake.

Update: 26.09.06:

code tested so far on Perl installations:

• MSWin32, perl 5.8.8, Windows XP
• GNU/Linux (gentoo), perl 5.8.8

package ExtUtils::ModsInstalled;
# Last modified: 26 Sep 2006 at 02:35 AM EDT
#-----------------------------------------------------------------#
# Inherit from ExtUtils::Installed and discover other branches to 
# add to the two subtrees searched for .packlist files.           
# Author: Soren Andersen <somian *AT* cpan -DOT- org>             
# Status: experimental (pre-beta)                                 
# License: As for Perl                                            
#-----------------------------------------------------------------#

use base 'ExtUtils::Installed';
use Config ();
use File::Find 'find';
require ExtUtils::MM;
require VMS::Filespec if $^O eq 'VMS';
use strict;
use warnings;

use vars qw/ @autoparents %Config $VERSION
   @Hardwired_in_Config $OurDbg $MSish /;
$VERSION = '0.01';
*Config = \%Config::Config;
$MSish = ($^O =~ /^((?i:MSWin\d?\d?)|os2|dos|mint)$/);
@Hardwired_in_Config = qw(sitearch vendorarch archlib);
$OurDbg = 0; # Turn off after testing any modifications

{
    no strict 'refs';
    @autoparents =
                   map { s{\\}{/}g if $MSish;
             ($^O eq 'VMS'
                   ? VMS::Filespec::unixify($_)
                   : File::Spec::Unix->canonpath($_))  }
           grep {$_} map($Config{"${_}exp"}, @Hardwired_in_Config);
}

my $onlyonce = { map (($_,1),@autoparents) };
unshift @autoparents, $_ for
             grep {$_ && !$onlyonce->{$_}++ && -d "${_}/auto"}
                 map { s{\\}{/}g if $MSish;
              ($^O eq 'VMS'
                   ? VMS::Filespec::unixify($_)
                   : File::Spec::Unix->canonpath($_)) }
         grep($_ ne '.',@INC);

if ($OurDbg) {
    printf STDERR "WARNING: %s was not tested with this more ".
       "recent release (%s) of ExtUtils::Installed\n",__PACKAGE__,
           $ExtUtils::Installed::VERSION if 
              $ExtUtils::Installed::VERSION > 0.08;

    print STDERR "Will look for .packlist files under:$/";
    print STDERR $_ for
    map { sprintf qq[%s\n   %s\n],$_, `du -sh "$_/auto"` } @autoparent
+s;
}

# Much of our constructor is just lifted verbatim from EU::Installed 0
+.08.
sub new {
    my ($class) = @_;
    $class = ref($class) || $class;
    my $self = {};

    my $archlib = $autoparents[$#autoparents];
    # Read the core packlist
    $self->{Perl}{packlist} =
      ExtUtils::Packlist->new( File::Spec->catfile($archlib, '.packlis
+t') );
    $self->{Perl}{version} = $Config{version};

    # Read the individual module packlists
    my $sub = sub {
        # Only process module .packlists
        return if $_ ne ".packlist" || $File::Find::dir eq $archlib;

    my ($fqmodpn, $module, $modname);
        $module = $File::Find::name;
    $module =~ s{\\}{/}g if $MSish;
    $fqmodpn = $module;

        # Behead the pathname & convert what is left to a module name
        for my $instloc (@autoparents) {
        $module=~ s!\Q${instloc}\E/auto/(.*)/.packlist!$1!s and
        $modname = $1 and last;
    }
    return unless $modname;

        my $modfile = "$modname.pm";
        $modname =~ s!/!::!g;

        # Find the namespace-top module file in the @INC pathlist
        $self->{$modname}{version} = '';
        foreach my $dir (@INC) {
            my $p = File::Spec->catfile($dir, $modfile);
            if (-r $p) {
                $modname = _module_name($p, $modname) if $^O eq 'VMS';
                $self->{$modname}{version} = MM->parse_version($p);
                last;
            }
        }
        # Read the .packlist
        $self->{$modname}{packlist} = 
          ExtUtils::Packlist->new($fqmodpn);
    warn "No .packlist for $modname?!" unless $self->{$modname}{packli
+st};
    };

    my(@dirs) = grep { -e } @autoparents;
    find($sub, @dirs) if @dirs;
    return(bless($self, $class));
}

1; # ARRET! (All Real modules must REturn True)
__END__

=head1 NAME

ExtUtils::ModsInstalled - like EU::Installed but discover
other branches to add to the two subtrees searched for .packlist files
+.

=head1 SYNOPSIS
  From the commandline:

 $ perl -MExtUtils::ModsInstalled -l \
    -e'my $in=ExtUtils::ModsInstalled->new;' \
    -e'push @mods,$_ for grep(/\Q$ARGV[0]/,$in->modules);' \
    -e'print for map{sprintf qq[\n%-56s %s\n].("-"x68).q[%s],$_,$in->v
+ersion($_),' \
    -e'join("$/   ","",$in->directories($_,"all"))} @mods' \
   Image


=head1 DESCRIPTION

As of version 0.08 the module C<ExtUtils::Installed> misses many
modules installed to the Perl system, it is reported. This class
inherits from ExtUtils::Installed and merely substitutes a different
C<new> constructor in order to cast a wider net (and also fix a few
small nits that to the author's tastes could be purged from
C<ExtUtils::Installed>'s "new()" code). If ExtUtils::Installed is
fixed in the future the need for a hack like this will go away.

Note that 

=head1 SEE ALSO

L<ExtUtils::Installed>, L<ExtUtils::Packlist>

=cut

(26.09.06) On one system the "SYNOPSIS" example code shown in the POD above resulted in this output:

Image::Magick                              6.2.9
--------------------------------------------------------------------
   /usr/lib/perl5/vendor_perl/5.8.8/i686-linux/Image
   /usr/lib/perl5/vendor_perl/5.8.8/i686-linux/auto/Image/Magick

Image::Size                              2.992
--------------------------------------------------------------------
   /usr/bin
   /usr/lib/perl5/site_perl/5.8.7/Image
   /usr/lib/perl5/site_perl/5.8.7/auto/Image/Size
   /usr/share/man/man1
   /usr/share/man/man3

[ysth]

Firstly, I don't see a config parameter called vendorarchexp. I am using the perl packaged with Debian. Maybe vendorarchexp is only set when perl has been built with vendor specifics. Here's the tail of my Config.pm:

Some time ago, most of the config stuff was moved out of Config.pm to save load time/memory. Only the most popularly inspected options are left. But you can still request $Config::Config{'vendorarchexp'} or just do perl -V:vendorarchexp.