Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

how to improve: use MODULE VERSION LIST

by smile4me (Beadle)
on Jan 07, 2017 at 00:49 UTC ( #1179107=perlquestion: print w/replies, xml ) Need Help??

smile4me has asked for the wisdom of the Perl Monks concerning the following question:

I curate a repository of cpan modules @WORK that is shared across the enterprise. Unfortunately, I am trying to serve two masters: "run the biz" and "change the biz". The RTB folks don't like changes to their environment, and fight vehemently when modules upgrade (b/c they have production systems dependent upon old code). By contrast, the CTB folks love the bleeding edge and want to try new things, build new tools, etc.

So, when I built Perl for the next operating system, I added a new directory to the default @INC where I can put additional, non-core, modules (or new versions of existing modules) for the CTB folks. This works fine if the "newer version" of a module comes earliest in the @INC list. But what happens when something in the corelist is updated? DHOH!

So after much GIYF, I found this very troubling statement:

"Please recall that Perl interpreter will STOP trying to search [additional directories in @INC] once it finds the file in one of the locations, without trying to see if the file is in later locations as well. E.g. if /usr/lib/perl/This/Here/MyModule2.pm exists, then Perl will not look for, nor care about the existence, of /opt/custom/lib/This/Here/MyModule2.pm." Stack Overflow

Is there a better solution than "use lib reverse @INC" to get the newer module that is further in the @INC list?

For example, autodie v.2.23 is in the core for perl. So I build the newest autodie v.2.29, and it now lives in the non-core extras modules path, which is part of the default @INC.

--> perl -MData::Dumper -E 'say "INC: ", Dumper( \@INC )' INC: $VAR1 = [ '/usr/foss/packages/perl/perl-5.20.3/lib/site_perl/5.20.3/x8 +6_64-linux-thread-multi', '/usr/foss/packages/perl/perl-5.20.3/lib/site_perl/5.20.3', '/usr/foss/packages/perl/perl-5.20.3/lib/5.20.3/x86_64-linux +-thread-multi-ld', '/usr/foss/packages/perl/perl-5.20.3/lib/5.20.3', '/usr/foss/packages/cpan-basics/perl-5.20.3/lib/perl5/x86_64 +-linux-thread-multi-ld', '/usr/foss/packages/cpan-basics/perl-5.20.3/lib/perl5', '/usr/foss/packages/cpan-basics/perl-5.20.3/lib/perl5/site_p +erl', '/usr/foss/packages/foss-perllib', '.' ]; --> perl -E 'use autodie; say q[version: ], $autodie::VERSION; say q[location: ], $INC{q[autodie.pm]}; ' version: 2.23 location: /usr/foss/packages/perl/perl-5.20.3/lib/5.20.3/autodie.pm --> perl -E 'use lib reverse @INC; use autodie; say q[version: ], $autodie::VERSION; say q[location: ], $INC{q[autodie.pm]};' version: 2.29 location: /usr/foss/packages/cpan-basics/perl-5.20.3/lib/perl5/autod +ie.pm --> perl -E 'use autodie 2.29; say q[version: ], $autodie::VERSION; say q[location: ], $INC{q[autodie.pm]}; ' autodie version 2.29 required--this is only version 2.23 at -e lin +e 1. BEGIN failed--compilation aborted at -e line 1.

So again, I ask: is there a better solution than "use lib reverse @INC" to get the newer module that is further in the @INC list? Is there a way to change the module search heuristic in Perl so that all the locations are scanned when you specify a VERSION (and/or a LIST)? Am I missing a new feature that solves this problem?

TIA for your wisdom and insights.

Replies are listed 'Best First'.
Re: how to improve: use MODULE VERSION LIST
by kcott (Bishop) on Jan 07, 2017 at 09:12 UTC

    G'day smile4me,

    It's not clear whether your "CTB folks" want the newer/additional modules for just their own use ("try new things, build new tools, etc.") or to build code that will subsequently become production code.

    If just for their own use, I'd suggest setting PERL5LIB (in .bash_profile, or equivalent) and that's all that needs to be done. From "perlrun: ENVIRONMENT":

    "A list of directories in which to look for Perl library files before looking in the standard library and the current directory."

    If production code, your "use autodie 2.29" example would be the better option.

    Manipulating @INC is fraught with problems. Reversing it, and you end up with '.' at the front of the list: not a good idea. More sophisticated manipulation, like moving the first four standard paths (site/non-site + arch/non-arch) with

    BEGIN { @INC = @INC[4 .. $#INC-1, 0 .. 3, $#INC] }

    would change the @INC you show to

    '/usr/foss/packages/cpan-basics/perl-5.20.3/lib/perl5/x86_64-linux-thr +ead-multi-ld', '/usr/foss/packages/cpan-basics/perl-5.20.3/lib/perl5', '/usr/foss/packages/cpan-basics/perl-5.20.3/lib/perl5/site_perl', '/usr/foss/packages/foss-perllib', '/usr/foss/packages/perl/perl-5.20.3/lib/site_perl/5.20.3/x86_64-linux +-thread-multi', '/usr/foss/packages/perl/perl-5.20.3/lib/site_perl/5.20.3', '/usr/foss/packages/perl/perl-5.20.3/lib/5.20.3/x86_64-linux-thread-mu +lti-ld', '/usr/foss/packages/perl/perl-5.20.3/lib/5.20.3', '.'

    but falls over if the base @INC ever changes (e.g. you add vendor_perl paths) or the environment changes (e.g. PERL5LIB or PERLLIB are set).

    — Ken

      Thanks kcott for the reply, I'll try to clarify:

      Once the CTB folks have added a new feature, using a new or improved CPAN module, they certainly want it to go into production. Thus, they are willing to update the code to:

      use autodie 2.29

      Where I run into problem is that Perl does not find version 2.29 given the order of the default @INC. Remeber, the RTB folks are still looking for autodie 2.23 in the core perl paths when the interpreter was compiled; but, the CTB folks want to get the new one. Unfortunately, when compiling Perl with the otherlibdirs option (to include additional locations), it adds the new locations to the end of @INC.

      So it seems, I am left with no other choice but to manipulate the @INC order in the code. Typically, once tools make their way into production, the owner (unix user) of scripts has a very limited shell environment (think apache user or www user). So relying on PERL5LIB, while okay for testing, is not so good for production.

      Thus, your slicing the INC is an option. BTW, why is the '.' location not good in the @INC?

      Smile
      Steve

        Thanks for the clarifications. To be honest, it sounds like your problems are more political than technological. You probably need to implement a plan for upgrading software that's agreed to by all parties: probably involving rigorous testing, appropriate notifications, sign-offs at various levels and so on. I'll leave you to work on that. See also codiac's comments.

        Here's a suggestion that doesn't require altering @INC directly or via the lib pragma, nor does it rely on checking version numbers. (I've used version numbers in the examples below purely for identification purposes, the solution itself doesn't need them.)

        Some up-front information:

        $ alias perle alias perle='perl -Mstrict -Mwarnings -Mautodie=:all -E' $ perl -v | head -2 | tail -1 This is perl 5, version 24, subversion 0 (v5.24.0) built for darwin-th +read-multi-2level

        Let's say you've got an xyz module. That's in production and lives in directory old. Here's old/xyz.pm:

        package xyz; our $VERSION = '1.0'; sub import { print "import called with args: @_\n"; } 1;

        You can use that with or without a VERSION; it might have a LIST, which could be empty.

        $ perle 'BEGIN { @INC = qw{old new .} } use xyz 1.0; say $xyz::VERSION +' import called with args: xyz 1.0 $ perle 'BEGIN { @INC = qw{old new .} } use xyz 1.0 (); say $xyz::VERS +ION' 1.0 $ perle 'BEGIN { @INC = qw{old new .} } use xyz 1.0 qw{a b c}; say $xy +z::VERSION' import called with args: xyz a b c 1.0 $ perle 'BEGIN { @INC = qw{old new .} } use xyz; say $xyz::VERSION' import called with args: xyz 1.0 $ perle 'BEGIN { @INC = qw{old new .} } use xyz (); say $xyz::VERSION' 1.0 $ perle 'BEGIN { @INC = qw{old new .} } use xyz qw{a b c}; say $xyz::V +ERSION' import called with args: xyz a b c 1.0

        Now, let's say that a new version of the xyz module becomes available. It has some features the developers want. You install it in directory new. Here's new/xyz.pm:

        package xyz; our $VERSION = '2.0'; sub import { print "import called with args: @_\n"; } 1;

        For the purposes of demonstration, that's identical to old/xyz.pm except for the version number. Attempting to use it by just specifying VERSION doesn't work because old/xyz.pm is found first in @INC (i.e. the problem already discussed).

        $ perle 'BEGIN { @INC = qw{old new .} } use xyz 2.0; say $xyz::VERSION +' xyz version 2 required--this is only version 1.0 at -e line 1. BEGIN failed--compilation aborted at -e line 1.

        What I suggest you do is create a new module that only needs to be very simple. I've called this Bleed::xyz and put it in the current directory (i.e. the last place in @INC to be searched). Here's ./Bleed/xyz.pm:

        package Bleed::xyz; our $VERSION = '2.0'; BEGIN { require 'new/xyz.pm'; } sub import { shift; xyz->import(@_); } 1;

        Now version 2.0 of xyz can be used with all the various combinations of VERSION and LIST.

        $ perle 'BEGIN { @INC = qw{old new .} } use Bleed::xyz 2.0; say $xyz:: +VERSION' import called with args: xyz 2.0 $ perle 'BEGIN { @INC = qw{old new .} } use Bleed::xyz 2.0 (); say $xy +z::VERSION' 2.0 $ perle 'BEGIN { @INC = qw{old new .} } use Bleed::xyz 2.0 qw{a b c}; +say $xyz::VERSION' import called with args: xyz a b c 2.0 $ perle 'BEGIN { @INC = qw{old new .} } use Bleed::xyz; say $xyz::VERS +ION' import called with args: xyz 2.0 $ perle 'BEGIN { @INC = qw{old new .} } use Bleed::xyz (); say $xyz::V +ERSION' 2.0 $ perle 'BEGIN { @INC = qw{old new .} } use Bleed::xyz qw{a b c}; say +$xyz::VERSION' import called with args: xyz a b c 2.0

        When version 2.0 of the xyz module is accepted into production, and assuming that involves moving new/xyz.pm to old/xyz.pm, you'll need one, very minor change to Bleed::xyz (i.e. s/new/old/). All existing code should continue to work. You can change instances of use Bleed::xyz to just use xyz at your leisure.

        "BTW, why is the '.' location not good in the @INC?"

        I didn't say that. I said:

        "... you end up with '.' at the front of the list: not a good idea.".

        Consider someone's who's bad. In /home/bad they write code with names like strict.pm, warnings.pm, autodie.pm, etc. that all do bad things. They then run your code from /home/bad. If @INC starts with '.', bad things will happen.

        — Ken

        > Unfortunately, when compiling Perl with the otherlibdirs option (to include additional locations), it adds the new locations to the end of @INC.

        Prepending the locations to @INC wouldn't help, as the RTB folks wouldn't be able to load the old version anymore. When searching for a module, Perl checks the version only when it finds the first implementation. If the version is less than the one specified, Perl dies, it doesn't look for a different version in the rest of the paths.

        $ cat My.pm package My; our $VERSION = "1"; warn "1 loaded"; 1; $ cat 0/My.pm package My; our $VERSION = "2"; warn "2 loaded"; 1; $ perl -I. -I0 -e 'use My 1; warn $My::VERSION;' 1 loaded at My.pm line 2. 1 at -e line 1. $ perl -I0 -I. -e 'use My 1; warn $My::VERSION;' 2 loaded at 0/My.pm line 2. 2 at -e line 1. $ perl -I. -I0 -e 'use My 2; warn $My::VERSION;' 1 loaded at My.pm line 2. My version 2 required--this is only version 1 at -e line 1. BEGIN failed--compilation aborted at -e line 1. $?=255 $ perl -I0 -I. -e 'use My 2; warn $My::VERSION;' 2 loaded at 0/My.pm line 2. 2 at -e line 1. $ perl -I0 -I. -e 'use My 1; warn $My::VERSION;' 2 loaded at 0/My.pm line 2. 2 at -e line 1.

        But, interestingly

        $ perl -I. -I0 -e 'eval q{use My 2}; warn $My::VERSION;' 1 loaded at My.pm line 2. 1 at -e line 1.

        ($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord }map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,
Re: how to improve: use MODULE VERSION LIST
by 1nickt (Abbot) on Jan 07, 2017 at 01:09 UTC

    You could:

    • Alter @INC at program compile time rather than perl compile time, so your CTB users do:
      use lib '/usr/foss/packages/foss-perllib';
      which unshifts your directory onto the front of @INC.

    • Edit: the following does not work, I was mistaken as shown by choroba and ikegami. (While there are no doubt gory reasons why it is so, it seems an uncharacteristic lack of DWIMery from Perl ...) Continue with the @INC you've compiled in, and instruct your CTB users to require a specific version, e.g.:
      use autodie 2.29;
      which ensures that perl will keep looking until it finds the specified version (in your directory).

    Hope this helps!


    The way forward always starts with a minimal test.

      That's not true. In order to determine the version of a module, Perl must first load it. As such, Perl can't load anything but the first module of a given name it find, even if a version if specified.

      $ cat Foo/Mod.pm package Mod; $VERSION = 1; 1; $ cat Bar/Mod.pm package Mod; $VERSION = 2; 1; $ perl -e'use lib qw( Foo Bar ); use Mod 2;' Mod version 2 required--this is only version 1 at -e line 1. BEGIN failed--compilation aborted at -e line 1.

      Thanks 1nickt.

      Yeah, I have been coaching folks to do the use lib 'new dir' technique, to rearrange @INC. Which works okay when used in combination with "use autodie 2.29".

      What bothers me most is that I have to manipulate @INC at all, especially when specifying a VERSION that I'm looking to use. The real problem is that Perl will not keep looking for the newer version of a module, if an older one is found earlier in the @INC list of locations.

      Smile,
      Steve

        That's not how it should be working.

        If you have a newer version of a module in a directory that you add with use lib, that should be the one loaded, without need of a version number (so long as that's the last change you make to @INC).

        You should be able to do:

        $ perl -Mautodie -E 'say $autodie::VERSION' 2.23 $ perl -I/usr/foss/packages/foss-perllib -Mautodie -E 'say $autodie::V +ERSION' 2.29


        The way forward always starts with a minimal test.

        As I mentioned in my previous post, I think it would be better to have separate installations of Perl, each with the required set modules of the required versions, with a development installation that is kept up to date. When a new application moves to production, make a new installation of Perl for it and install the modules it needs. this way, each application can have the version of Perl and whatever modules it needs, without worrying about the needs of other applications.

Re: how to improve: use MODULE VERSION LIST
by codiac (Beadle) on Jan 07, 2017 at 23:12 UTC
    IMO you are solving the wrong problem :) The problem is you have two environments (for now :-)) and you are storing & deploying them together. This can only end in tears when change X for team Y affects team Z and you have no idea why Team Z's production app suddenly exploded when no one touched it.
    1. Separate the module versions in your repository, paths, branches, however your repo is set-up.
    2. Use jails, chroots, containers, VMs, etc, to isolate the deployments.
    3. Only expose the appropriate repository version to each deployment.
    Personally I'd just use your OS's package manager for the modules as it makes isolating deploying & upgrading individual systems easier to orchestrate if the environment starts to scale out ... but I also prefer cider to beer so YMMV ;)
      Personally I'd just use your OS's package manager for the modules

      Are you suggesting the OP rely on the OS's package repository? Or suggesting the OP create packages for deployment using the OS's package manager?

      If the first, generally it is a bad idea for "non system" Perl applications to use the "system Perl" as updates are more likely to cause problems. It is better to leave the system Perl for OS's use and install one or more separate versions of Perl for non system applications.

      If the second, I think it would be easier (and more portable) to deploy applications by creating the same type of package CPAN uses. The resulting tar.gz file can be copied to each server and the cpan command con be used to install it:

      $ cpan cpan> install myapp.tar.gz

      Would even be possible for the OP to setup a CPAN-like archive and direct cpan to use that before using the actual CPAN archive.

Re: how to improve: use MODULE VERSION LIST
by ikegami (Pope) on Jan 09, 2017 at 14:32 UTC

    I would install two addition builds of Perl. This would allow RTB to maintain the specific version of modules with which they are comfortable, and it will allow CTB to upgrade modules without breaking the system.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1179107]
Approved by Paladin
Front-paged by ww
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (1)
As of 2021-08-03 01:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My primary motivation for participating at PerlMonks is: (Choices in context)








    Results (32 votes). Check out past polls.

    Notices?