A Vision for Easy Web Application Deployment for Perl

[markjugg]

The problem

Installing a Perl web application should be as easy as "1. Unpack 2. Upload 3. Visit URL for configuration". Typically it isn't this easy, especially when the application follows the best practice of re-using CPAN modules.

By contrast, my average experience installing a PHP web application has been impressively easy, approximating the three step process I outlined above.

There's certainly nothing about the language itself that restricts this ease use. DadaMail has basically got it.

The DadaMail difference is that bucks the norms of CPAN culture. It doesn't exist on CPAN, and it bundles the modules it depends on from CPAN rather than requiring the users to fetch them themselves.

CPAN was designed with geeks in mind, not end users.

Having end users get modules themselves from would a better experience if it were designed for them. As an installation cpan appears to primary desired for users who run as root and have a shell and and compiler handy. Sure, there are documented ways to install modules as a non-root user, but it Shouldn't Be That Hard.

As a result, Perl apps aren't seeing the user base that they could. For example, selenium-regen is a nice helper script for use with Selenium. It comes bundled in the WWW-Selenium-Utils distribution. It would be useful to a lot of people outside of the Perl community, but fussing with installing related CPAN modules is required to use it.

The merits of bundling prequisites

Some people react negatively to the idea of bundling prerequisites, as DadaMail has done. When you do this, the argument goes, it will be difficult to deploy upgrades of the bundled modules because you now have several copies in use with one copy in each bundle.

Doing sysadin for a website development and hosting company, I've had the chance to experience the recommended "everyone uses one copy" philosophy. I'll be clear this has significant drawbacks as well. This design assumes that upgrades tend to make things better and that backwards compatibility is maintained. In practice, the only thing for certain about an upgrade is that Something Changed. The more projects that depend on that module, the greater chance there is that one them depended on the behavior before the change happened.

So while this approach may save module installation time, it potentially destabilizes the overall system, and requires quality assurance testing of all the projects that use the required module. Attention may still be needed to be given to many projects hosted on the server.

A little infrastructure may go a long way.

I have a vision for an alternate way to access CPAN that addresses these concerns.

This site would be designed for end-users and end user applications, not necessarily programmers and modules. However, unlike all the "script archives" out there, this system would be designed for projects built primary with CPAN modules.

When a user clicks a download link here, they would receive a tar ball which bundles all the distributions the project depends on. The distribution authors would only need to generate hints about these in a normal CPAN upload. The infrastracture would take care of generating the tar balls.

The result would be something that could be unpacked and uploaded, without additional installations or command line fussing.

Handling Perl XS

Handling Perl XS code would be the trickiest part of this system. "XS" code is C code compiled for a particular architecture.

Here we can follow in the footsteps of how Linux distributions already handle this for Perl XS code: by making binary packages for each target architecture.

From my initial poking around, it seems we have the key tools to address this systemically, by automatically (or easily) generating and uploading platform-specific binaries for each distribution which directly uses XS.

PAR generates platform-specific binaries for Perl distributions. Adding PAR integration to a typical Perl distribution is so easy that it could be automated in many cases. For a typical Makefile.PL, you can simply use inc::Module::Install; instead of ExtUtils::MakeMaker, and then add a line at the bottom like:   par_base('MARKSTOS');. Now make par is available to generate a PAR file made just for your platform.

In Conclusion

Making Perl applications easier to deploy helps us all. Our work becomes more accessible to others, lowering the bar for gaining new users and contributors.

Historically, The infrastructure of CPAN has been a tremendous assest our community. Let's take the lesson of the value of good infrastructure and use it to expand the reach of Perl.

[dragonchild]

Several mostly-unrelated comments:
• Perl6 will address the XS issue by assuming you have Parrot installed. Since Parrot does all the heavy-lifting, nearly every CP6AN distro should be "pure-perl".
• The primary issue isn't "How do I install this as a non-root user?" but "How do I install this in a place that my code will find?" You are generally expected to be root because the perl executable has root-only paths hard-coded. It seems to me that we need to do two things:
  1. Provide more documentation/tutorials/whatever to get "use lib" into more use.
  2. Provide an easier way in CPAN(PLUS) to install into a directory on a per-install basis. Maybe something like "install Foo into lib" or "installdir lib\ninstall Foo" or something like that. But, configuring CPAN as a non-root user when there's already a root user requires source-diving and crazy configuration. If you're not a root user, it shouldn't use the root config.
• Building on the prior comment and using Dadamail as an example, Dadamail should be able to specify to CPAN(PLUS) that it wishes to install all the prerequisites into a given directory, assuming that the prereqs aren't already there. You should also be able to specify to Dadamail where your personal installs are, if you wish. Then, Dadamail should be able to build a string of "use lib" items that it places in a special .pm file that it sources in immediately, probably through 'do' because 'use' scopes too much. That kind of work needs to be automated.
• The XS issue in Perl5 is a tough one, especially as most interesting apps will want to use MySQL and DBD::mysql is problematic in a shared-hosting environment. You either compile against 4.1+ or pre-4.1 and the two cannot live together, just like MP1 and MP2 cannot live together. Maybe C-compilers are just a requirement with free ones listed (gcc, MingWcc, etc).
• The interface to CPAN needs improving. The three .gz files that get downloaded shouldn't be that way by default. Whenever I go to cpan, I'm usually issuing 1-3 "install Foo" commands and that's it. cpan should be able to ask some CGI front-end "What's the correct FTP command for grabbing the Foo module?" and let a centralized database do the rest. Here's why - I often want to build my own personal CPAN so that I can use CPAN.pm to update my company's modules on production instead of using SVN. That's just one more thing to install on production machines and I don't like that. (The alternative is bundling and installing by hand which is fraught with danger as befits any by-hand solution.) If the API for CPAN was a clear and simple solution (with maybe a personal CPAN app?), then doing this would be a piece of cake. Imagine that as your upgrade solution for your company's modules!

[markjugg]

Thanks for all the feedback. I want be clear that the "cpan" or "cpanplus" shouldn't be required tools for an end user to have, in part because they shouldn't need to use or understand a shell.

If tools like they are involved at all, they should come into play as part of the web-based configuration step. There the software can ask permission, and then install extra modules under the covers.

[dragonchild]

Sure, if that's how you want things to go. Just embed a webserver and let it do its thang.

[tirwhan]

Ugh, I'm sorry, but having each application install its own version of common libraries in its own location sounds to me like a maintenance nightmare. What happens if there's a security bug in an underlying module which makes it necessary for you to upgrade it? You then have two options:

1. Find every instance of that module and update it in place, making sure that you break none of the applications in the process.
2. Wait for all applications to release a new version with the upgraded module, which can take days to months, leaving you vulnerable in the meantime.

Both of these are extremely nasty. Additionally, encouraging the practice of using "private" versions of modules will lead to brittle applications which assume they know exactly which kind of environment they're working in, as well as a plethora of "tweaked" versions of modules for each app.

The CPAN way works extremely well for the two more common usage scenarios, that of a multi-user machine maintained by a (team of) admins whose responsibility it is to check that upgrades do not break apps (this can be helped a lot by using an OS/distribution which does sane dependency checking and provides clean upgrades for the included packages), and that of a single-user machine where the admin is also the developer/user. It does not work quite so well (as you have illustrated) for the shared web hosting environment where untrusted users need to install their own modules. Frankly, I don't consider that usage scenario important enough to make things harder for the rest of us. Maybe some specialized applications which are most commonly used in webhosting environments could go this way, but I'd recommend against making this common practice.

As an addendum, there is a single PHP web app I use, and I run multiple instances of it. Every time a security-related bug is discovered somewhere in the package I need to upgrade each and every instance of the app instead of just one underlying library, which pisses me off no end.

[markjugg]

What happens if there's a security bug in an underlying module which makes it necessary for you to upgrade it?

You have disqualified yourself from the target user if have multiple versions of the same things install, and you have a clue about security.

1. Find every instance of that module and update it in place, making sure that you break none of the applications in the process.

The part about "making sure you break none of the applications" is required step for all applications that use the module, whether you upgrade it one place or multiple places. And that's where most of the work. The "find every instance" would be a simple "find", if you didn't already know by memory or from documentation.

2. Wait for all applications to release a new version with the upgraded module, which can take days to months, leaving you vulnerable in the meantime.

This could be solved be the infrastructure. Remember the "auto generated packages"? When a new module is published to CPAN the package depends on, a new bundle could be available immediately. (Perhaps because the bundles would be generated on the fly or with smart caching). Like Linux distro packaging systems, the bundle name might include a revision in addition to the version, indicating that the bundled prereqs have been updated.

Additionally, encouraging the practice of using "private" versions of modules will lead to brittle applications which assume they know exactly which kind of environment they're working in, as well as a plethora of "tweaked" versions of modules for each app.

Remember the design? The distributions would be regular CPAN distributions with regular dependencies. I'm just talking about generating an additional alternate view for end users. Since the authors are CPAN uploaders, they won't suddenly lose a clue about good software practices.

Frankly, I don't consider [the shared web hosting] usage scenario important enough to make things harder for the rest of us.

Ouch. This attitude of not caring about end users may do something to explain why several PHP applications now at the top of my list for features and end-user ease-of-use, even though I would prefer a Perl solution: osCommerce, PhpPgAdmin, PhpBB, PhpAds, WordPress, Drupal.

The net effect is that I'm now learning some PHP to light customizations to osCommerce and Drupal, because as a pragmatic user, they already have extensions written for them that do about everything I want.

Unless I've missed something in my research, Perl's got some catching up to do in the end-user killer web-app category, and I think some infrastructure focused on that could really help.

The result would actually be better, because in the background we would be using many of the well-documented, well-coded and well-tested we're already familiar with from CPAN.

[tirwhan]

You have disqualified yourself...

I'm sorry, I don't understand that sentence, care to explain what you mean here?

..."making sure you break none of the applications" is required step...whether you upgrade in one place or multiple places

Yes, but it's much harder, because you've got dog-knows how many different versions of the module installed for the different apps (that's the whole point of this scheme after all, right, to be able to have the version of a module which the app is written against) and you now have to find out what changes between each of those versions and the newly fixed one. Much, much harder. And yeah, finding the module file is easy enough, but then you have to care enough to upgrade the gazillion instances you have strewn all over your disk, and time being a scarce resource this will often not happen. This is a prime recipe for increasing the number of out-dated Perl modules installed on production machines and giving Perl a bad security rap.

..auto-generated packages..

I'm confused, if you can auto-generate the packages you're presuming that the bundle will work with the latest version of the module. So what's the difference to installing the latest module version into a central place? The only difference I can see is a false sense of security on the part of the user who thinks "this must work because I downloaded the Bundle" and will be all the more disappointed if it breaks.

..won't suddenly lose a clue about good software practices

Anything that gets made easier will be done more often. If module authors no longer need to care about their app working with the latest version of an underlying module version they will be tempted to use deprecated features, module internals etc., because "it works if you use the distribution". Other things (like adding the latest feature) will take precedence over robustness. That is part of my point, keeping different versions of modules on the same machine should be something that requires manual intervention, it's not good practice (most of the time) and you should have to think about what you're doing.

This attitude of not caring about end users

You misunderstand, I do care about the end user, and installation of a software package is indeed an important factor in its uptake (I've more than once refrained from trying out a Java app because of the horribly horrible horror that is Java installation and package handling). But Perl isn't a one-trick pony like PHP, it has applications that go far beyond the web-hosting sector. The requirements of all of these users needs to be taken into account when trying to improve some aspect of the language, not just of a relatively small subset of the user base.

If someone wants to use PHP instead of Perl (or Ruby or Python or any other language) and has good reasons for doing so, more power to them. To me the disadvantages of PHP are numerous enough that I'll prefer Perl any day, and one of these is the area of security. I care much more about Perl getting a bad reputation for its security than I care about the convenience of amateur webmasters (just to be clear, I'm not saying all users who use shared webhosting are amateurs, but those for whom this particular issue is a problem probably are).

I do agree that dependency handling an installation from CPAN is not as end-user friendly as it could be, but this is a very hard problem to solve correctly. I don't see any merit in auto-bundling software together for the purpose of spewing multiple versions of module all over your harddisk.

[gunzip]

You're banging your head against a brick wall, I'm afraid. The Perl community doesn't appear to care less when PHP leaps ahead in popularity with web developers. See "What sets Perl Back" which I posted in Meditations a couple of weeks ago. CPAN is Perl's killer app and also it's Achilles heel. CPAN.pm and CPANPLUS.pm have a long way to go before they can be called end-user apps. You have to ask why there is only one book on Perl and MySQL (Dubois) and why it wasn't reissued - because there wasn't enough interest in the first edition. The book's brilliant but a lot of the Perl community seem to think web development is beneath them.

[itub]

having each application install its own version of common libraries in its own location sounds to me like a maintenance nightmare.

Having to worry that some of the applications you have installed will break whenever you upgrade any module is also a maintenance nightmare. In the end, you should be able choose which kind of nightmare you prefer, ;-) which may depend on your situation. A user who just wants to install one blog application is not in the same situation as a webmaster maintaining several applications.

A good compromise could be to provide both options: a standard CPAN package that lets you install the dependencies separately wherever you want, and a "user-friendly" bundle that you can just unwrap and works out of the box.

[Aristotle]

A good compromise could be to provide both options:

Exactly. I wonder why noone else brought this up. Why do we always have to ask either-or questions? Often, there’s no problem pursuing both alternatives.

[perrin]

This is also how Krang works -- prereqs are bundled, and you can either build from source with the included build script or build a binary distribution for a specific platform. The modules are installed locally, which means, among other things, you can have several versions of Krang installed on one machine for testing purposes, even if they require different versions of the same CPAN module.

[tphyahoo]

I completely agree there is a problem. As for your solution, I have to spend more time rereading it before forming an opinion but... yes. Major problem.

I spent Christmas working on the documentation for installing catalyst on a shared server. It is *not* easy. It needs to be, or this technology just won't get adopted and will die. The catalyst approach for this is, use CPANPLUS... but CPANPLUS, though better, is still not that user friendly.

What my gut wants is, default installations for CPAN and CPANPLUS without having to answer all those questions, working 90% of the time, on a shared hosting environment. Because that's the reality, lots of people use shared hosting. I was able to make the CPAN install pretty painless by accepting the defaults and editing Config.pm after the fact, but I wasn't able to do this for CPANPLUS.

I guess the bottom line is, all those CPAN/CPANPLUS install questions are intimidating to people who just want to start developing, and they will switch to other solutions if this doesn't get streamlined.

[rinceWind]

markjugg, thanks for this. Automatic bundlement on the hoof through infrastructure - I like it! Are you aware that you can have multi-architecture PAR balls, which might be a better fit than platform specific ones?

You might be interested in the slides for a talk I have given at YAPC::Europe 2005 and the London Perl Workshop.

[Booger]

I'm not going to bother offering my opinions on this topic since others have already provided several intelligent replies however I would like to state my experience as it applies to this issue.

The company that I work for develops and supports a web app (and some other associated technologies) that is used throughout the newspaper industry. We develop and support this app in-house and distribute it to customers who run their own copies on their servers & internal systems. We're not a hosting company. Our cash-flow comes from the customization & support of this application to suit individual workflows; we don't make money from the actual app per-se.

Our development team makes heavy use of the more mature CPAN modules (and a few which aren't quite as mature) and ran into deployment issues related to CPAN usage early on in our project.

Our final solution was to package the required modules (and whatnot) along with our application and to develop an installation system that automatically compiles & installs pre-requisites in a sub-directory of our chosing (i.e. /opt/oursoftware-1.0).

It isn't perfect but it's a whole lot easier for our customers to manage (not to mention our in-house teams) and has huge benefits from a support perspective. We require of our customers a particular OS and provide a customized operating environment for our applications. This allows us to customize the various pieces of middleware while avoiding all of the sticky issues that arise from having to deal with software installed by another vendor. A shared environment for applications is wonderful but only in situations where you have complete control the environment. Our software often runs on dedicated servers but there are some situations where we have to share server space with other vendors and believe me when I say that you do NOT want to be responsible for downtime in a newspaper setting, particularly when you are involved with pre-press ad management.

[vek]

Been a while since I've looked but I seem to recall that Movable Type does it the same way. No need for the end user to ever install modules via a shell as all common non-core libraries are bundled in $INSTALL_DIR/extlib.

[markjugg]

Moveable Type is a great example of a high-profile Perl web app that has a reputation for being easy to install. I'm not surprised they use a similar approach.

[qq]

I saw a very impressive talk at YAPC::EU Braga about RunApp. It attempts to address these problems, and looked like it did a good job.

I have not tried it myself, as this isn't actually a pressing need. I'd recommend the talk as the module docs look a little sparse

[skazat]

Heya,

I'm the author of Dada Mail, the app used as an example.

I thought I'd give a quick, "how do I do" for bundling the CPAN modules into Dada Mail.

As markjugg pointed out, the audience for Dada Mail is more likely to be your casual website owner, small business person, etc - not a fellow developer. As such, these people may know how to sling some HTML to a very good degree and will know how to configure fairly complex web apps that have simple configurations, like Dada Mail, Moveable Type, etc but WILL NOT be able to easily jump into a shell and fire up CPAN. It's also one more step in setting up a program and with this audience, too many steps leads to your application being passed by for a different alternative. Some examples of similar applications that I have been told are too hard to either use or to install are Gnu Mailman and actually a php application, called PHPList. I haven't installed either myself.

Anyways,

I keep a record of the CPAN modules I need for Dada Mail in a CPAN Bundle - Here it is: http://cvs.sourceforge.net/viewcvs.py/*checkout*/mojomail/dada_mail_stable/dada/extras/developers/DadaMail.pm

It's in a CVS repository, so any changes, of course, will be annotated to a ridiculous degree. Any other security advisories about the program would come from the regular sources. These are usually few and far between.

Since the Bundle file is POD, it's also used to make documentation: http://mojo.skazat.com/support/documentation/DadaMail.pm.html

When I make a distribution of Dada Mail, I'll run cpan myself and get a fresh copy of all the needed modules and test with this setup. If something breaks, I'll know about it before it gets released. Just to clarify: the only thing in CVS is the Bundle File - no actual CPAN modules are kept in CPAN - always plucked fresh for each release (each release seems to be now about every 2 - 4 weeks)

As for, "What happens if there's a security problem". If there's a change in a CPAN module, the new version of Dada Mail will automatically have the fix and any testing done will see if anything breaks. (I don't have a test suite, but it's in the pipeline)

Fixing this type of problem basically means reinstalling the entire distribution of Dada Mail again. If you're a little slicker and know how to use cpan and have root access, you can basically just make a custom distribution of Dada Mail for your specific environment by removing the, "perllib" directory that comes with Dada Mail and running the DadaMail.pm bundle locally. This following scenario is mentioned in the POD of the Bundle file itself. I don't see this being any more dangerous than allowing a user to have their of person perllib in the home directory.

I think there's an added benifit of bundling your own CPAN modules with your application and that is: different version of CPAN modules do act differently, as markjugg pointed out. For example, Dada Mail relies on the HTML::FromText CPAN module, but can only use version 1.005 of the module and nothing later. This is because the 2.x version of the module needs a much larger CPAN module requirement itself, some of these modules being XS modules, which, as a rule, isn't something I attempt to require, since the target audience won't know/doesn't have the correct resources to compile code.

What markjugg is dreaming of may be actually easier to handle if it was the responsibilty of the app distributor (for the time being). I strung my current build tools with a simple shell script and a slightly custom CPAN config. I talk about how I did this here: CPAN Shell, Bundles and downloading specific versions and here: Install a CPAN module at x location

There has been an instance of Dada Mail using a CPAN module and customizing the CPAN module itself to make it actually work very different than before. This CPAN module is now not a prereq and isn't a part of the Bundle. I don't see this as a problem, as it's sort of the whole idea of Free Software: you put something out because you think it's useful, allowing people the freedom to use and modify it as they see fit.

Just quickly glancing at the Bundle file for Dada Mail, you'll notice that it relies on a fair amount of CPAN code - about 3.2 megs. This includes some big ones, like MIME-Tools. I would say the rest of the app is about .5 megs. Dada Mail would not be even close to what it is now without CPAN. Dada Mail is very far in its life as a program and any program of this complexity will have problems dealing with all the different pieces. Knowing exactly which CPAN modules are being used in each version greatly simplifies making sure the app you ship will work. I can bet you the popularity of the program would not be as great as it is now if the CPAN modules were not installed automatically, or, if I only used the core modules.

Hope that enlightens a bit.

[techcode]

As someone who is making money to pay for the studies by freelancing (as in writing web apps using Perl) I agree with Mark.

And what I do is exactly that - bundle all those modules that are used with the code that I wrote. But unfortunately that is only part of the problems that Perl for Web is facing :(

Even that technique still requires that the end user is messing up with some *.cgi or *.pl files to set up the shebang line ... and to chmod it. So I usually install the applications myself. As I use CGI::App ;) it's only one file that dispatches to modules.

The other great problem is performance. CGI is out because of it performance problems, yet mod_perl still isn't quite suitable for any "Joe" on shared hosting. Mod_php still does it much better. I admit, it doesn't give you the power that mod_perl does - but Joe doesn't care ...

Sure there are other solutions like FastCGI or _____ (just put whatever you like/use here). But do web servers come with that as default? I don't think so - so who (which administrator) would even bother to install that? Especially when php does the job - and is mostly used anyway ...

I believe that Ruby is facing similar performance (CGI vs mod_ruby) problems when it comes to hosting.

When we solve those problems - we can start thinking about better advocating Perl ...

Let's first see what others are doing good and how they are doing it. Then let's adopt it ...

[pajout]

Just my experience:
For some reasons, i am trying to develop own web application framework, running under Apache/mod_perl. My idea is not to wrap Apache module nor force some templating schema, but to support easy usage of mod_perl features without detail knowledge about everything of mod_perl and to support possibility of code reusability.
I like clarity and simplicity, and I tried to use h2xs to build standardized perl project. Perhaps I am ignorant, but I did not successive, specially, when I did need to include some non-perl directories.
So, I have my devel directory, potentially synchronized with CVS or SVN repository, including perl modules and configurations and example applications. No autoconf, automake, reconf, Makefile.PL or another building/installing stuff. One row have to be included into httpd.conf (to include my piece of apache configuration) and apache must be restarted. Thats all. Other configurations are described in a framework conf and application confs, which are in the xml format.
I am writing this note to imagine, how it is easy to develop, deploy and/or use some software - do cvs checkout, include one row into the service configuration and restart the service. I know that I have the very specific constraints - I assume installed Apache/mod_perl and I am not dependant on other modules yet...

[westernflame]

I agree that expanding the reach of Perl on low level hosting is an excellent idea. However, you suggestion of creating tar balls does not do this. It is the same old approach to creating web applications. You have a series of scripts that rely on modules. The user has to set permissions, edit configuration files, use ftp etc. To much of these things are beyond the basic user. Creating custom tar balls will not solve this.

An alternative approach would be to have a standardised dispatch script for using perl modules as web applications. It would be a single script that specified a module, a method in that module, and query arguments. All web applications would have to be a module, and could all be installed from CPAN.

To make such an approach viable for people with no Perl skill, an alternative interface to CPAN could be devised. One that didn't require shell access. Such a system would need to be able to operate entirely through the browser, downloading modules in segments, as a page refreshed. Alternatively if no web access modules where available (such as LWP), a wizard could be created to help users upload modules themselves.

With a pair of standardised scipts (dispatch and install), CPAN could become an excellent way of distributing web apps.

[markjugg]

I had something much like this in mind and agree that is a great idea. Both the CGI::Application and Catalyst frameworks have some dispatching built in, allowing you to use a single script for that. The CGI::Application is currently revamping CGI::Application::Dispatch to be even more powerful in this regard.

I'm not sure what you mean by the "alternative interface" to CPAN, because that's the core of what the idea is. It provides an interface to CPAN that generates tarballs which are easy to unpack and upload without shell access. For fancy configurations, a web-based installation wizard might well talk to CPAN directly to install additional stuff.