Beefy Boxes and Bandwidth Generously Provided by pair Networks RobOMonk
XP is just a number
 
PerlMonks  

Maintaining an Enterprise Perl Distribution

by cbrandtbuffalo (Deacon)
on Mar 24, 2005 at 19:04 UTC ( #442165=perlquestion: print w/ replies, xml ) Need Help??
cbrandtbuffalo has asked for the wisdom of the Perl Monks concerning the following question:

We use Perl widely in my shop--enough so that I consider it deployed on an enterprise scale. However, as time has gone by, we've gotten woefully out of date on our version of Perl and our module versions. One of the blessings of open source is also one of the drawbacks: there is no vendor to force you to upgrade. We've reached a point where we've decided to bite the bullet and get everything up to date.

I'm looking for some wisdom from the Monks on how to keep our distribution up to date after we upgrade everything.

Some specific questions:

  • How often do you upgrade your Perl? What version do you run in your production environments?
  • How often do you upgrade your modules? Do you try to always stay with the latest version?
  • How do you deploy your Perl distributions? (install modules on each machine, autodeploy somehow, etc.)
  • Do you include external packages (like external C libraries) in your distribution system?
  • Do you track what modules are actually being used? Do you remove modules that no one is using anymore (one less module to monitor and stay on top of)?
  • How do you test your applications after a module or Perl upgrade?

My current proposals for our system are something like this:

  • We'll upgrade to the latest stable, then stay within two releases of stable.
  • We'll run upgrades of all modules every six months. We won't upgrade to a version that is less than a month old (I've seen too many releases followed by quick bug-fix releases.). We will have provision for interrim upgrades of a module if necessary. We'll use the CPAN module with some scripts to manage this process.
  • We currently use our Perforce version control system to manage and deploy our distribution using scripts. This works well, but we don't currently track external dependencies (for example, expat for the XML modules).
  • For testing, we have all developers run their tests, including WWW::Mechanize tests for our web apps. This can be a bit time-consuming, so that's why I'm going to suggest only twice per year.

I'm also interested in any other advice on maintaining a reasonably up-to-date Perl distribution in an enterprise environment.

Thanks!

Comment on Maintaining an Enterprise Perl Distribution
Re: Maintaining an Enterprise Perl Distribution
by dragonchild (Archbishop) on Mar 24, 2005 at 19:38 UTC
    The very first thing you need to figure out is

    Do I need to upgrade at all?

    Upgrading is a change. It's a massive change, actually. Is the cost of testing this change worth the benefits? What are the benefits? Do you have a backout plan if something breaks?

    Personally, I would upgrade on a case-by-case basis. If version 2.44 of the Floopy module works for you, why does it matter if the latest version is 4.55? If you never hit a bug, then you don't need the bugfixes. Plus, bugfixes might introduce bugs you're not aware of. If you need a feature in 4.55, then upgrade to it.

    Furthermore, the issue isn't bugs within a given module - it's bugs between modules. Integration testing is expensive! And, testing doesn't prove the abscence of bugs - it simply shows you haven't found any yet.

    Thinks for a second . . .

    The biggest problem I have with your plan is that you're changing too many things at one time. Change one item, then see what happens. If nothing breaks, bake it in for a month. Then, change another thing and see what happens. Lather, rinse, repeat.

    Being right, does not endow the right to be rude; politeness costs nothing.
    Being unknowing, is not the same as being stupid.
    Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
    Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

      That's a good point, and one I'm laboring over.

      Things are working OK now, but there are a few 'little' things wrong with some of our modules that are fixed in newer releases. Some of these modules, like Class::DBI, have many dependencies, so an upgrade involves updating other modules.

      The problem with standing pat is it's hard to get help when things are broken. When we have an error, the first thing the Monks or people on a list ask is "what are you running?" Too often in the recent past, we have this strange mix of modules and versions. So the answer is "upgrade and see if that fixes it."

      And this is a completely legitimate answer since it is just too hard to track interdependencies across strange combinations of module versions.

      So I guess I'm hoping we can better troubleshoot things by staying more current. But maybe this isn't the case?

      Another solution might be to just do spot upgrades when we have to because we run across something. The problem here is exactly what you mentioned: testing is hard and costly. If we just did it 'as needed', it would also be un-planned. We have a lot of perl code running, so it's difficult to arrange for ad hoc testing of everything. If we had an upgrade schedule, at least we could plan the testing.

      I agree wholeheartly with many things you said here. Change isn't always a good thing. I strongly recommend using a test server to see what breaks when you upgrade anything before you just deploy it all over, *ESPECIALLY* if you are highly dependant on it.

      In my personal experience as a system administrator, unless mandated by the need for a feature, repair of a bug or a security update, an upgrade is an unneccesary act and often times a very dangerous one. When I was new to system administration, I thought that it was important to stay "within two releases" etc as is mentioned in the initial post. I quickly realized (after a horrible problem surfaced) that my logic may have been a bit flawed. I adopted a new philosophy...upgrade when there is some need to do so, otherwise, leave it alone.

      dragonchild is right about changing too many things at once as well. If something does break (and trust me...it WILL), it will be far more difficult to find and correct the problem if you changed multiple things. If you *must* upgrade many things, do it in reasonable phases where you can simplify any troubleshooting. You will thank yourself later.

      Just my $0.02.


      perl -e 'print reverse qw/o b n a e s/;'
Re: Maintaining an Enterprise Perl Distribution
by perlfan (Deacon) on Mar 24, 2005 at 19:54 UTC
    * How often do you upgrade your Perl? What version do you run in your production environments?

    I move up whenever the need behooves me. Most machines I work on are still at 5.6.1. More importantly, I try to do things in a way that minimizes any breakage that might occur. This means, usually, that I try to use features available only in the lowest common denominator, which for me is 5.6.1.

    * How often do you upgrade your modules? Do you try to always stay with the latest version?

    I am one of those who tries to stick as close to core as possible, so when I do need a module, I install it. I usually do not have everything and the kitchen sink installed. Every once in a while, I check out the latest versions of the modules I use, and only upgrade if there is a compelling reason to do so.

    * How do you deploy your Perl distributions? (install modules on each machine, autodeploy somehow, etc.)

    Modules, usually. http://par.perl.org is also a good thing to look into if you are depolying apps to a customer.

    * Do you include external packages (like external C libraries) in your distribution system?

    http://par.perl.org will let you do this, but I again am one of those who tries to use pure Perl with as few dependencies as possible.

    * Do you track what modules are actually being used? Do you remove modules that no one is using anymore (one less module to monitor and stay on top of)?

    No, but I use the modules that install - i.e., I only install modules when I need them.

    * How do you test your applications after a module or Perl upgrade?

    I don't really - unless it *breaks*.

    * We'll upgrade to the latest stable, then stay within two releases of stable.

    As far as your plan goes, it is difficult to say what you should do. Do what your obligation to the customer demands. The best thing for you to do, in my opinion, is to keep current with info on the modules that you use, upgrade only if there is a compelling reason to do so, and test its effects in test environment before you blindly deploy your app that relies on untested (to you) third party software.
Re: Maintaining an Enterprise Perl Distribution
by LanceDeeply (Chaplain) on Mar 24, 2005 at 21:27 UTC

    are your client/user/development machines out of sync with each other? it seems part of your problem is geting a standardized environment. support can be very hard if everyone is running different versions of xyz module.

    once your shop can agree on a standard environment, answering the other questions should be easier. upgrading for the sake of upgrading has kicked my ass more than once, so get your dev crew together and find out what the needs are.
    once you guys agree on the standard, you'll have some fun getting the dev group upgraded/standardized. once that mess is sorted, you'll have the experience needed to tackle this on your end user machines, production servers, etc...

    hth
      We're in sync across our environments, so we don't have that problem right now. We've got dev, QA, and prod servers, and several of each. We also have other servers where we run perl and we've just gotten those in sync since we started deploying from version control. When we put in changes, we test at each level and roll forward if things are good.
Re: Maintaining an Enterprise Perl Distribution
by jhourcle (Prior) on Mar 24, 2005 at 21:33 UTC

    I've had systems running versions of perl that were years out of date. This has only been an issue in two types of situations --

    one of the developers (okay, me), needed some functionality that was only in a new version.
    (I think this happened once in 9 years.)
    Solution: Install the new version of perl in an alternate location. (see the discussion in Solaris, Perl 5.8 and use 5.005) Developers who want to use this new version would need to explicitly specify this new version.
    Problem: may not work for mod_perl; this is a pain if you're just doing it for a module, so you don't muck with the original one in place
    Problems moving scripts between systems.
    Okay, this one has stung me a few of times. Sometimes it's just a different perl location in the shebang line, but there have been a couple of times when my code didn't work with more recent versions of the module. (I remember a bad time once with Net::LDAP, but I just went and backed down the version so the systems matched, and I didn't have to change my code, as it was being moved because of a system failure, and I didn't have time to spend on it). Personally, I just look at which modules are giving problems, and fix those issues, rather than going for a complete sync across the board.

    So, basically, to answer your specific questions:

    1. I don't, if I can avoid it. The installed default, and possibly whatever was current at the system was configured.
    2. When I need some new functionality that the new module provides ('functionality' to include bug fixes). No.
    3. Install modules as needed, on a per machine basis.
    4. nope, as I do it by hand.
    5. System documentation. No.
    6. I try not to upgrade, but if I have to, I run whatever tests the development teams have given me. Which is to say, I don't run any tests. But I then leave it running on the development servers until management signs off on it being tested, and then move it to production. (If something breaks, I've got record of someone signing off in the change management system, showing they didn't test their stuff).

    Yeah, I've probably read too much BOFH over the years, but well, I've been a developer, and a sysadmin ... if it's important, you write tests for it. If management drives a schedule that results in nothing being thoroughly tested, I make sure they sign off the system before upgrades.

    Note: I've since changed jobs, to where I only have to deal with one production system (that's in my control... we have four replicas, distributed geographically, on hetereogeneous versions and hardware, but I don't manage them), and I get control of my part of the production schedule, within reason, so I don't have nearly so many issues anymore. It's much better than trying to deal with ~30 web, ldap and mail servers and idiot management.

Re: Maintaining an Enterprise Perl Distribution
by Qiang (Friar) on Mar 24, 2005 at 23:04 UTC
    this is a topic comes up quite oftenly. I would like to see a summary on this issue in the tutorial section. ( I don't know that much on this to do this job ;)

    but to get it start, here is another discussion not long ago. Smooth perl upgrades

Re: Maintaining an Enterprise Perl Distribution
by JSchmitz (Canon) on Mar 25, 2005 at 00:53 UTC
    The CPAN autobundle can be used to ease perl upgrades, by creating a special bundle containing all the installed modules of the current version of perl. This bundle can then be installed once the new version of perl is installed.

    cpan> autobundle


    Wrote bundle file
    /var/spool/cpan/Bundle/Snapshot_2004_10_01_00.pm


    cpan> quit

    # upgrade perl here

    # cpan
    cpan> install Bundle::Snapshot_2004_10_01_00
Re: Maintaining an Enterprise Perl Distribution
by brian_d_foy (Abbot) on Mar 25, 2005 at 05:42 UTC

    Didn't we run into a problem with a module that was upgraded and changed behaviour on us? :)

    Before you upgrade, you can use your development servers to make sure everything works with the new versions, so that's already a leg up on what most people have.

    I definitely recommend staying away from anything that smells like early-adopter. I tend to think that old code does the same thing that it always did, and if old code is working and you have other things to think about, other things should win. Other people in Stonehenge are at the other end of the spectrum. :)

    --
    brian d foy <bdfoy@cpan.org>
      Yes, we did. I think it was the URI module. It changed how it translated some things in the uri_escape method.

      This is one of my bigger fears. Module authors can change behavior, or even just change a default setting. If we have code that relied on this default, it might break. Actually, I think DBI changed a default in a somewhat recent update and I've got it on my list as something that might break.

      The safe thing here is to check for the know changes before the update, but I don't relish going through the change notes for every module we have installed to see if they changed a key behavior or setting. But if we want to know what we're getting into, this may be necessary.

      I have a script to report what code is using what module. For modules that only have one user, I guess I could get that developer to look into things. My problem there is that too often I was 'that developer' who recommended using a module. :)

Re: Maintaining an Enterprise Perl Distribution
by jerlbaum (Sexton) on Mar 28, 2005 at 04:35 UTC
    Hi cbrandtbuffalo --

    Break your requirements into two:

    1. System which happen to use Perl, such as UNIX servers which perform some other task.
    2. Servers which host production Perl applications.

    The first situation is very easy to deal with: Upgrade Perl when you upgrade your OS. Consider the out-of-the-box RedHat Enterprise server. It comes with Perl, and has its own mechanism for keeping Perl up to date. If you don't have custom Perl apps on that server, let RedHat's up2date handle it for you. When you upgrade from ES3 to ES4 you will get whatever version of Perl comes with the latter.

    The second case can be handled as follows:

    1. Recognize that this is a development issue and not an operations issue. IOW, it's controled by your developers' need.
    2. Dev should set the Perl version on a per project basis.
    3. Build your apps with all their libraries local. An *excellent* example of this is the Krang CMS project.

    If you adopt the methodology used by Krang, you will find that you are immune to upgrade problems, and have a very solid platform for testing new versions of Perl on your apps before you put them into production.

    TTYL,

    -Jesse-

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://442165]
Approved by kutsu
Front-paged by kutsu
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (6)
As of 2014-04-20 10:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (485 votes), past polls