Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

ExtUtils::Installed doesn't list all modules

by rinceWind (Monsignor)
on Sep 24, 2006 at 18:51 UTC ( #574615=perlquestion: print w/replies, xml ) Need Help??

rinceWind has asked for the wisdom of the Perl Monks concerning the following question:

I have a need for trawling the existing Perl code that has already been installed. Fine, it looks on the surface that rather than hand rolling something, ExtUtils::Installed with its modules method should do the trick.

Sadly, I find that EU::I is missing huge swathes of modules on my system. Diving into the code, I find that it only looks in two places: $config{archlibexp} and $config{sitearchexp}. Why oh why, I ask myself, just these special places? Surely the code should be traversing @INC. Heck, you could even have custom directories added via use lib or $ENV{PERL5LIB} that you want to be visible.

Maybe at the time EU::I was crafted (crufted?) there were assumptions about perl directories that could be relied on. This is clearly not the case now, and I am wondering: should this be raised as a bug? Also, how much legacy cruft might be relying on this behaviour?

I see that gellyfish has been down this road before. However, I am reluctant to patch locally ExtUtils::Installed, as it is bound to get reverted at some future point in time. Has anything been formally presented? What was the reaction?

So, I'm looking at hand-rolling something that does the business. Unless there's another candidate module out there that I'm unaware of (please tell me). I intend to build this generically and release it to CPAN.

--

Oh Lord, won’t you burn me a Knoppix CD ?
My friends all rate Windows, I must disagree.
Your powers of persuasion will set them all free,
So oh Lord, won’t you burn me a Knoppix CD ?
(Missquoting Janis Joplin)

Replies are listed 'Best First'.
Re: ExtUtils::Installed doesn't list all modules
by jacques (Priest) on Sep 24, 2006 at 19:04 UTC
    You can use my module: HTML::Perlinfo::Modules. The latest version was just released yesterday (unfortunately there are some POD errors I need to fix asap.)

    If you want to see if a module is installed, you can simply do:

    use HTML::Perlinfo::Modules; my $m = HTML::Perlinfo::Modules->new(); my $module = $m->print_modules(show_only=>qr/Rince::Wind/); if ($module) { print "Module Rince::Wind is installed!"; } else { print "No Rince::Wind module found"; }
    If you want to see all of the modules installed, you simply say:
    $m->print_modules;
Re: ExtUtils::Installed doesn't list all modules
by davidrw (Prior) on Sep 24, 2006 at 19:11 UTC
    Surely the code should be traversing @INC. Heck, you could even have custom directories added via use lib or $ENV{PERL5LIB} that you want to be visible.
    ...
    So, I'm looking at hand-rolling something that does the business. Unless there's another candidate module out there that I'm unaware of (please tell me).
    There are several different existing approaches ..

      My researches and supersearches before I posted my question showed me your list of related threads, and this was useful. However, evidence would seem to suggest that there isn't an ideal solution already out there.

      I am not interested in HTML formatting for my needs, just a raw list of module names. Similarly, I am not interested in a command line script.

      If EU::I did what it says on the tin, that would have been it. I guess I'll roll my own.

      --

      Oh Lord, won’t you burn me a Knoppix CD ?
      My friends all rate Windows, I must disagree.
      Your powers of persuasion will set them all free,
      So oh Lord, won’t you burn me a Knoppix CD ?
      (Missquoting Janis Joplin)

        Well, how about a hash ref whose keys are the names of all the modules in @INC? Would that meet your needs? This isn't in the documentation for my module, but you could do:
        use HTML::Perlinfo::Modules; my $hashref = (HTML::Perlinfo::Modules::find_modules("",1,\@INC))[1]; # Here are all the module names in @INC: print "$_\n" for (keys %$hashref);
        P.S. If you want even more information on those modules you can look into the values of those keys. Or just use Data::Dumper to see what's in that hashref.
Re: ExtUtils::Installed doesn't list all modules
by Intrepid (Deacon) on Sep 25, 2006 at 04:35 UTC

    rinceWind wrote (originally at 13:51 GMT +5 on 24 Sept 06):

    Sadly, I find that EU::I is missing huge swathes of modules on my system. Diving into the code, I find that it only looks in two places: $config{archlibexp} and $config{sitearchexp}. Why oh why, I ask myself, just these special places? Surely the code should be traversing @INC. Heck, you could even have custom directories added via use lib or $ENV{PERL5LIB} that you want to be visible.

    I've just looked at the code for ExtUtils::Installed and I have some answers. Not the asked-for solutions, but some answers. It's become de rigeur to bash ExtUtils:: generally (?). My opinion is: let's at least be correct when we do.

    Why, I ask myself, just these special places? Surely the code should be traversing @INC. The code is traversing @INC in its search for the module names found via the mechanism I will describe shortly. See line 116 in the ExtUtils/Installed.pm shipped with perl-5.8.8 by ActiveState, for example (that's 0.08). Or refer to the code to follow my explanation if you don't have an editor or other Web tab open on the source already. See the new() subroutine specifically.

    The new sub defines two vars as below:

    my $archlib = $Config{archlibexp}; my $sitearch = $Config{sitearchexp};
    and IMHO these two vars need to be joined by a third var, $Config{vendorarchexp}. That's a flaw in the module under almost any reasonable scope of expectations that I can conjure in my mind. However, it is not unreasonable that these two existing Config vars are used as they are, and this use is not limiting the scope of the search in exactly the manner explained by the OP. The reason for examining only FOOarchexp is that the values associated with each are the locations under which .packlist files are found.

    The .packlist files (familiar to those of us who watch the actions of module installers (ExtUtils::MM, Module::Build) closely) were installed when each module was added to the Perl installation. ExtUtils::Installed is using the presence of these .packlist files as a means to the end of listing what modules have been installed.

    Edit 25.09.06: correct next statement

    It does not read the .packlist file in order to locate the module, you'll notice (because you are really reading the code by now, right? -- because it's more interesting than this long-winded explanation); ... the code simply takes note of the path to each (under .../auto/ ) and then creates a module name from that pathname. This is pretty reasonable behavior actually (in my analysis) in that it correlates well with how Perl's long-established mechanisms for how module management works.

        Soren A / somian / perlspinr / Intrepid

    -- 
    Words can be slippery, so consider who speaks as well as what is said; know as much as you can about the total context of the speaker's participation in a forum over time, before deciding that you fully comprehend the intention behind those words. If in doubt, ask for clarification before you 'flame'.

      Thanks for the insights. I have been digging a bit further.

      ... and IMHO these two vars need to be joined by a third var, $Config{vendorarchexp}. That's a flaw in the module under almost any reasonable scope of expectations that I can conjure in my mind. However, it is not unreasonable that these two existing Config vars are used as they are, and this use is not limiting the scope of the search in exactly the manner explained by the OP. The reason for examing only FOOarchexp is that the values associated with each are the locations under which .packlist files are found.

      The .packlist files (familiar to those of us who watch the actions of module installers (ExtUtils::MM, Module::Build) closely) were installed when each module was added to the Perl installation. ExtUtils::Installed is using the presence of these .packlist files as a means to the end of listing what modules have been installed.

      Firstly, I don't see a config parameter called vendorarchexp. I am using the perl packaged with Debian. Maybe vendorarchexp is only set when perl has been built with vendor specifics. Here's the tail of my Config.pm:

      tie %Config, 'Config', { archlibexp => '/usr/lib/perl/5.8', archname => 'i486-linux-gnu-thread-multi', cc => 'cc', d_readlink => 'define', d_symlink => 'define', dlsrc => 'dl_dlopen.xs', dont_use_nlink => undef, exe_ext => '', inc_version_list => '5.8.7 5.8.6 5.8.4 5.8.3 5.8.2 5.8.1 5.8.0', intsize => '4', ldlibpthname => 'LD_LIBRARY_PATH', libpth => '/usr/local/lib /lib /usr/lib', osname => 'linux', osvers => '2.6.15-1-686', path_sep => ':', privlibexp => '/usr/share/perl/5.8', scriptdir => '/usr/bin', sitearchexp => '/usr/local/lib/perl/5.8.8', sitelibexp => '/usr/local/share/perl/5.8.8', useithreads => 'define', usevendorprefix => 'define', version => '5.8.8', };

      Secondly, I can think of several use cases where .packlist files are living elsewhere. The one that is biting me here is that I am on a system which has had its Perl version upgraded. The machine was originally Debian Sarge, which comes with 5.8.4, and is now Etch, with 5.8.8. Here's my @INC:

      /etc/perl /usr/local/lib/perl/5.8.8 /usr/local/share/perl/5.8.8 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl /usr/local/lib/perl/5.8.4 /usr/local/share/perl/5.8.4

      And I find that my .packlists are evenly divided between being under /usr/local/lib/perl/5.8.4 and /usr/local/lib/perl/5.8.8. I don't believe I have a broken perl installation.

      Consider the user installing modules not as root. They don't have write permissions to any of the perl lib directories, so they set PERL5LIB and install to their own private directory tree, packlists and all. This is the scenario for me at work; I am a developer, not a sysadmin.

      Finally, it occurs to me that the packlists might not be giving the whole picture, even if we are picking up all of them. We know that current versions of ExtUtils::MakeMaker and Module::Build respect and honour packlists, but Module::Build has only done this relatively recently. I've also found cases of Module::Build installs where the module author has hijacked the install mechanism, and there's no packlist in sight. And, what's to say about other package management schemes. ActiveState PPM probably respects packlists, but I believe it's up to the module packager for Debian dpkg, Redhat RPM, yum, etc. to include the .packlist (or not, as might be the case).

      --

      Oh Lord, won’t you burn me a Knoppix CD ?
      My friends all rate Windows, I must disagree.
      Your powers of persuasion will set them all free,
      So oh Lord, won’t you burn me a Knoppix CD ?
      (Missquoting Janis Joplin)

        I'm preparing my swing (a module which will be my effort to address this problem you've raised. I'm just posting this now as a message and a 'placeholder' to make myself obligated to posting my effort ;-). This node will be updated later.

        Update: 25.09.06: added all below.

        OK. While not perfect (and I am a wee bit hurried in making this posting even), there's a solution to follow, YMMV, etc etc. We don't want to locally modify ExtUtils::Installed for reasons rinceWind so cluefully mentioned (a future upgrade could well just clobber our modifications someday). So let's create a new module and stick it in the sitelib where (hopefully) no harm will come to it.

        Since ExtUtils::Installed is doin' the OO mambo, let's dance with it. We'll just make our own constructor (which ordinarily is called just once per run). See how this works for you, rinceWind.

        Update: 26.09.06: CODE CHANGE -- found a regexp mistake.

        Update: 26.09.06:

        code tested so far on Perl installations:

        • MSWin32, perl 5.8.8, Windows XP
        • GNU/Linux (gentoo), perl 5.8.8

        package ExtUtils::ModsInstalled; # Last modified: 26 Sep 2006 at 02:35 AM EDT #-----------------------------------------------------------------# # Inherit from ExtUtils::Installed and discover other branches to # add to the two subtrees searched for .packlist files. # Author: Soren Andersen <somian *AT* cpan -DOT- org> # Status: experimental (pre-beta) # License: As for Perl #-----------------------------------------------------------------# use base 'ExtUtils::Installed'; use Config (); use File::Find 'find'; require ExtUtils::MM; require VMS::Filespec if $^O eq 'VMS'; use strict; use warnings; use vars qw/ @autoparents %Config $VERSION @Hardwired_in_Config $OurDbg $MSish /; $VERSION = '0.01'; *Config = \%Config::Config; $MSish = ($^O =~ /^((?i:MSWin\d?\d?)|os2|dos|mint)$/); @Hardwired_in_Config = qw(sitearch vendorarch archlib); $OurDbg = 0; # Turn off after testing any modifications { no strict 'refs'; @autoparents = map { s{\\}{/}g if $MSish; ($^O eq 'VMS' ? VMS::Filespec::unixify($_) : File::Spec::Unix->canonpath($_)) } grep {$_} map($Config{"${_}exp"}, @Hardwired_in_Config); } my $onlyonce = { map (($_,1),@autoparents) }; unshift @autoparents, $_ for grep {$_ && !$onlyonce->{$_}++ && -d "${_}/auto"} map { s{\\}{/}g if $MSish; ($^O eq 'VMS' ? VMS::Filespec::unixify($_) : File::Spec::Unix->canonpath($_)) } grep($_ ne '.',@INC); if ($OurDbg) { printf STDERR "WARNING: %s was not tested with this more ". "recent release (%s) of ExtUtils::Installed\n",__PACKAGE__, $ExtUtils::Installed::VERSION if $ExtUtils::Installed::VERSION > 0.08; print STDERR "Will look for .packlist files under:$/"; print STDERR $_ for map { sprintf qq[%s\n %s\n],$_, `du -sh "$_/auto"` } @autoparent +s; } # Much of our constructor is just lifted verbatim from EU::Installed 0 +.08. sub new { my ($class) = @_; $class = ref($class) || $class; my $self = {}; my $archlib = $autoparents[$#autoparents]; # Read the core packlist $self->{Perl}{packlist} = ExtUtils::Packlist->new( File::Spec->catfile($archlib, '.packlis +t') ); $self->{Perl}{version} = $Config{version}; # Read the individual module packlists my $sub = sub { # Only process module .packlists return if $_ ne ".packlist" || $File::Find::dir eq $archlib; my ($fqmodpn, $module, $modname); $module = $File::Find::name; $module =~ s{\\}{/}g if $MSish; $fqmodpn = $module; # Behead the pathname & convert what is left to a module name for my $instloc (@autoparents) { $module=~ s!\Q${instloc}\E/auto/(.*)/.packlist!$1!s and $modname = $1 and last; } return unless $modname; my $modfile = "$modname.pm"; $modname =~ s!/!::!g; # Find the namespace-top module file in the @INC pathlist $self->{$modname}{version} = ''; foreach my $dir (@INC) { my $p = File::Spec->catfile($dir, $modfile); if (-r $p) { $modname = _module_name($p, $modname) if $^O eq 'VMS'; $self->{$modname}{version} = MM->parse_version($p); last; } } # Read the .packlist $self->{$modname}{packlist} = ExtUtils::Packlist->new($fqmodpn); warn "No .packlist for $modname?!" unless $self->{$modname}{packli +st}; }; my(@dirs) = grep { -e } @autoparents; find($sub, @dirs) if @dirs; return(bless($self, $class)); } 1; # ARRET! (All Real modules must REturn True) __END__ =head1 NAME ExtUtils::ModsInstalled - like EU::Installed but discover other branches to add to the two subtrees searched for .packlist files +. =head1 SYNOPSIS From the commandline: $ perl -MExtUtils::ModsInstalled -l \ -e'my $in=ExtUtils::ModsInstalled->new;' \ -e'push @mods,$_ for grep(/\Q$ARGV[0]/,$in->modules);' \ -e'print for map{sprintf qq[\n%-56s %s\n].("-"x68).q[%s],$_,$in->v +ersion($_),' \ -e'join("$/ ","",$in->directories($_,"all"))} @mods' \ Image =head1 DESCRIPTION As of version 0.08 the module C<ExtUtils::Installed> misses many modules installed to the Perl system, it is reported. This class inherits from ExtUtils::Installed and merely substitutes a different C<new> constructor in order to cast a wider net (and also fix a few small nits that to the author's tastes could be purged from C<ExtUtils::Installed>'s "new()" code). If ExtUtils::Installed is fixed in the future the need for a hack like this will go away. Note that =head1 SEE ALSO L<ExtUtils::Installed>, L<ExtUtils::Packlist> =cut

        (26.09.06) On one system the "SYNOPSIS" example code shown in the POD above resulted in this output:

        Image::Magick                              6.2.9
        --------------------------------------------------------------------
           /usr/lib/perl5/vendor_perl/5.8.8/i686-linux/Image
           /usr/lib/perl5/vendor_perl/5.8.8/i686-linux/auto/Image/Magick
        
        Image::Size                              2.992
        --------------------------------------------------------------------
           /usr/bin
           /usr/lib/perl5/site_perl/5.8.7/Image
           /usr/lib/perl5/site_perl/5.8.7/auto/Image/Size
           /usr/share/man/man1
           /usr/share/man/man3

        Firstly, I don't see a config parameter called vendorarchexp. I am using the perl packaged with Debian. Maybe vendorarchexp is only set when perl has been built with vendor specifics. Here's the tail of my Config.pm:
        Some time ago, most of the config stuff was moved out of Config.pm to save load time/memory. Only the most popularly inspected options are left. But you can still request $Config::Config{'vendorarchexp'} or just do perl -V:vendorarchexp.
Re: ExtUtils::Installed doesn't list all modules
by demerphq (Chancellor) on Sep 27, 2006 at 09:00 UTC

    Send me a patch to rectify the modules behaviour and ill look into it. Im a little busy with other projects right now to do more than that. Im sorry.

    ---
    $world=~s/war/peace/g

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://574615]
Approved by GrandFather
Front-paged by Old_Gray_Bear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (6)
As of 2019-11-19 18:46 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Strict and warnings: which comes first?



    Results (95 votes). Check out past polls.

    Notices?