Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Smooth perl upgrades

by saintmike (Vicar)
on Dec 15, 2004 at 18:32 UTC ( #415142=perlquestion: print w/replies, xml ) Need Help??

saintmike has asked for the wisdom of the Perl Monks concerning the following question:

Fellow monks, has anyone worked through seamless perl upgrades on large systems?

If there's a large number of different Perl applications on a box, all using #!/usr/bin/perl (which is 5.00503) and I want to upgrade to 5.8.6, obviously I can't just replace #!/usr/bin/perl by perl 5.8.6, because that might break certain scripts only happy with perl 5.00503.

So, during a transition period, I need to support both /usr/bin/perl (5.00503) and, say, /some/other/path/bin/perl which is 5.8.6.

Old scripts are unchanged with #/!usr/bin/perl, new scripts will be using #!/some/other/path/bin/perl.

So far so good. Now let's look at the modules. Both installations are going to use different lib paths, so in order to install a new module on the box (like a security fix of a CPAN module), it needs to be installed for both perl5.00503 and perl5.8.6 (lets hope it works for both of them).

Now let's say that perl 5.10.0 becomes available. I'll add yet another perl install with its own lib directory, and now have to potentially upgrade three installs with every new module.

The problem is obviously that there's applications which no one wants to touch and therefore old perl installs have to be maintained (almost) indefinitely, slowly transitioning into a maintenance nightmare.

Has anyone in the monastry worked through transition periods like these and successfully managed them without going crazy?

Replies are listed 'Best First'.
Re: Smooth perl upgrades
by Old_Gray_Bear (Bishop) on Dec 15, 2004 at 20:22 UTC
    We did a transation from 5.6.0 to 5.8.2 a while back. The basic process was:
    1. Install new Perl in a distict set of libraries. We installed a basic set of CPAN modules in the separate libraries as well. We built the list by looking at the heavy-hitters (run often, company's bottom line depends on it, etc) and grabbing the modules they needed
    2. Published to the World that the New Perl was out there and that They Should Give It A Try. "If you have any missing modules, let us know and we will pull them from CPAN." People ran their codes from a command-line invocation using the explicit name (/usr/local/bin/perl5.8.2 -I /usr/local/lib....) and reported back on the oddities.
    3. Got Feed back, Added additional Modules to the list. Got more feed-back.
    4. At the end of the month, we sent out a Note to the World: "We will begin phasing over to 5.8.2 over the next three months. Here is the proposed order of march...."
    5. We waited for two weeks to get more feed-back on the proposal and then started.
    6. We changed the she-bang line to point to the 5.8.2 libraries as the first step of the conversion. For an awful lot of our code, that was the only change needed. For the couple or three programs that needed it, code changes were made and a new name assigned (SysA and SysA.5.8.2, for example). (Note: all code went into CVS; if we needed the old version, we had it. The new code was checked in on a separate branch; and *all* maintenance was done twice. A lot of the development groups instituted a partial freeze for the quarter.)
    7. As soon as all modules in a system were converted and tested, the new code was put into production.
    8. At the end of the 'Conversion' period we found we had touched about 85% of our code base. (About 6,000 packages and stand-alone scripts in almost 200 distict systems.) The remaining code fell into the following buckets:
      1. The script was no longer in production, just no one had noticed.
      2. It was a one-shot and would need to be severely breathed on before we used it again. (A Database fixup routine, for example.)
      3. The Owner told us that "It can't break, I coded it to avoid all of that fancy stuff." (Honest. We got *that* in writing. Good thing, too.)
      4. Orphans -- no one was admiting to be the Party Responsible. Management decided to assign ownership in a couple of cases. For the others, we took the 'wait and see' position.
    9. The actual 'conversion' occured on a Monday in the middle of the month, and in the middle of a billing cycle. We renamed the old Perl libraries to .../perl.5.6.0 and changed the symlinks to point to the new kids. We picked Monday to give us the maximum amount of time to sort things out before the weekend.
    10. There were a couple of hits right out of the box, but no real show-stoppers. They were fixed and the rest of the week ran cleanly.
    11. The next week we got a shot at the billing period-end codes. One or two more 'issues' cropped up, again nothing that a ten minute patch to the she-bang followed by a  perl -c couldn't handle.
    12. A month later, we ran through the quarter-end codes; same thing, most were OK, there were a couple of hiccups, which we fixed and went on.
    13. After the quarter-enders had been fixed, a note went out: "Here are the programs that have not been looked at the the Conversion. After this date(xx/xx/xxxx), any conversion work we do will have to have a Project Account to charge to."
    14. On The Date we:
      1. Archived the 5.6.0 Perl and libraries with a six month retention period.
      2. Did a mass CVS checkout and ran a code to alter the she-bang line, returning it to "/usr/local/bin/perl", and then merged everything back into the the CVS head.
      3. Did all this on a Saturday; the only time we had to have folks work outside of normal business hours.
      4. We Waited.
      5. At the end of the week, we declared it a Wrap and went out to the Local Pub for a celebration. Management bought, and was slightly horrified by the amount of beer four people can consume and still be able to hit a triple on a dart-board.
    15. There were still a few things that needed addressing later. (It turned out that the Accounting Folks had been mis-led about the Year-End codes being just a special run of the Quarter-enders that we keep for ten years rather than one). But by and large, things went cleanly.
    In summary, we wandered through 6K Pieces of Perl (averaging 300 lines per program/module) in a little over four months (four people, normal working hours, except for one weekend at the end), and ended up still talking to one another. The usual caveats about mileage and performance apply.

    I can't address the 'going crazy' part (there are those who say that I started out crazy, so....).

    I Go Back to Sleep, Now.


Re: Smooth perl upgrades
by bluto (Curate) on Dec 15, 2004 at 20:02 UTC
    While I understand there are times when you can't get around it, I really hate maintaining legacy perl trees. This tends to work ok if the machine's environment is relatively static. The problem I've seen with this is that as you upgrade the machine's OS, C compiler, libraries, etc then legacy code becomes more and more brittle and can eventually break in some hard to detect/fix way. Worse, if you need that upgrade to keep your newer code running smoothly, you now have a dilema (e.g. downgrade and workaround it; keep the upgrade, reinstall the old perl tree, and hope it works).

    I tend to compromise on this by keeping two versions of perl -- the vendor's ancient perl (/usr/bin/perl); and my own in it's own tree where I symlink to it from /usr/local/bin/perl. Any script that uses /usr/local/bin/perl tacitly agrees it will work with newer versions of perl. When I rebuild perl in a new tree, I make sure the modules in the old tree are installed in the new one as well. This allows me to move the tree easily to similar machines once it's configured properly.

    Of course, in order to actually verify that the older code works with a newer perl, it needs to be easily testable. This is where a test suite is crucial, and may actually be worth adding to your legacy code if you don't already have one. If it's not possible, you may be able to make a similar policy that you only support perl versions x and y and that any new code must have a test suite.

Re: Smooth perl upgrades
by Luca Benini (Scribe) on Dec 15, 2004 at 19:02 UTC
    I see 3 ways:
    1) your script start with #!/usr/local/bin/my_right_perl_bin
    2) put your perl in a path like /opt/perl/X.X.X/bin/ So if need 5.4.1 you can run it using
    PATH=/opt/perl/5.4.1/bin/:$PATH ./
    WARNING: your script must start with #!/usr/bin/env perl
    If the example above doesn't work try to use me@mypc$ set +h
    3) NOT GOOD WAY you can create a link /usr/bin/perl -> /opt/perl/X.X.X/bin/perl and change it, the problem with this solution is that the selected version is system wide

Re: Smooth perl upgrades
by Anonymous Monk on Dec 15, 2004 at 21:03 UTC
    I usually don't have to deal much with the code that is using the perl, so my techniques are more geared to maintaining the trees. Whenever I build a package (yes *every* time), I build it with a version specific directory.
    (or something) Then the active perl is created with a symlink
    /usr/site/perl -> perl-5.8.0
    Finally, individual binaries are linked in /usr/local/bin
    /usr/local/bin/perl -> /usr/site/perl/bin/perl
    Now if I want to create a new perl, I build it in a standalone environment.
    Anyone that wants to can simply try this version out by using a /usr/site/perl-5.8.2/bin/perl in their program. When the time comes, we announce that the active version will change. Anyone that needs to use it can add an explicit version to their script. Then we activate it.
    rm /usr/site/perl ln -s perl-5.8.2 /usr/site/perl
    Most of the time, there's no additions/subtractions required for the links in /usr/local/bin. More work on the front end (especially the links into /usr/local/{bin,lib,..}), but very handy later. Now most scripts simply make use of the new code, while those with other requirements can easily use the old one.

    In the past I've tried sharing modules between versions, but I've generally found it easier to simply rebuild all the modules for the new version. Costs a little diskspace.

Re: Smooth perl upgrades
by spq (Friar) on Dec 15, 2004 at 21:48 UTC

    Many excellent points have been made for your consideration. My personal habit is to intall the versioned binaries, and give people a few weeks warning of the upgrade so they can test. At the given date, I relink /usr/bin/perl to the new binary.

    Users can edit their own scripts either making whatever changes to migrate are required or changing their #! line to point to the about to be deprecated version. We keep our systemwide Perl programs in a single directory, which makes it a bit easier to copy the entire set, change the version they use, and test. Programs we want to deprecate we point to the old binary and don't bother testing.

    As for modules, the CPAN module provides a way to define your own bundle of modules available from CPAN. IIRC, this can even be done somewhat automtically. This can save a lot of effort on upgrades, since you can install your entire bundle from the latest versions directly off CPAN. I've generally opted for the more conservative approach of maintaining a directory where I store the tarballs of every Perl module I've installed on the system, so I can re-install the same versions as were currently in use into the new Perl versions tree.

    Best of luck!

      if you break it they will come!

      seriously just do it, but be ready to put it back at short notice. Maybe risky or maybe I'm just too trusting of perls backwards compatabilities:)

      you could even do it on a dev box first and test your apps a bit

      would be handy if someone had a script to check scripts for perl version compatability issues

Re: Smooth perl upgrades
by Anonymous Monk on Dec 15, 2004 at 18:57 UTC
    I've never had a "seamless" upgrade, but when I upgrade perl I rename it to its version. eg. perl560 and perl586. /usr/bin/perl is then a symlink to the latest version I have installed. The lib paths are different, so other than security module updates, typically you will only have to update the latest version's libraries. Legacy apps don't tend to need new libraries installed.
      The lib paths are different, so other than security module updates, typically you will only have to update the latest version's libraries.

      Your use of the word "only" suggests that this is a minor task. It often isn't minor at all, particularly if you have scripts laying around which rely variously on any of hundreds of CPAN modules.

      "My two cents aren't worth a dime.";
Re: Smooth perl upgrades
by ysth (Canon) on Dec 15, 2004 at 21:12 UTC
    Configure the new perl(s) with -Dversiononly. That way, you get not only a versioned perl executable (e.g. perl5.8.6), but a perldoc5.8.6, perlbug5.8.6, etc.

    Then you need to choose between installing all modules for each version, or sharing (to the extent possible by binary compatibility).

    I'm not sure where the maintainence nightmare comes into it; either you need to keep old perls around, or you don't. If you have old scripts depending on an old version of perl, you wouldn't typically touch that version at all, would you?

Re: Smooth perl upgrades
by shockers (Acolyte) on Dec 16, 2004 at 16:47 UTC
    If you have to have multiple version of perl on the same box (which I do), you can point to the one you need in your PATH.

    This is easy if you have multiple versions of perl on a windows box -- just set the PATH statement.

    It's relatively easy to do in UNIX as well. In the perl script, use:

    #!/bin/env perl

    and set your PATH accordingly.

    I'm writing scripts in a large company with multiple sites targeting thousands of machines running Win32, Linux, Solaris 8-9-10, Solaris 2.6, Solaris 2.51, SunOS 4.1.3. Login accounts and perl are not standardized corporate-wide, so I'm at the mercy of dozens of SA's who only worry about their location.

    Some boxes have /usr/bin/perl (pointing to /bin/perl) but not /usr/local/bin/perl and vice-versa. Not foolproof, but I set paths to both and the "#!/bin/env perl" will find at least one.

    But in the case of thousands of target machines running different OSs needing to run the same scripts but with no control over the SAs, we place the perl binaries of different OS's on a server. Logging in (using specific application ids) on a target machine would set/ensure the PATH points to the correct OS's perl on the server.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://415142]
Approved by Corion
Front-paged by Corion
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (6)
As of 2019-11-19 07:57 GMT
Find Nodes?
    Voting Booth?
    Strict and warnings: which comes first?

    Results (94 votes). Check out past polls.