On the scaleability of Perl Development Practices

[jdrago_999]

Monks,

In developing Perl web applications, we've all seen the same problems and/or anti-patterns arise again and again. Here I will point some of them out and hope for thoughts on possible solutions.

Too Many Options

I argue that Perl suffers from "Too much of a good thing." How many web development frameworks are there? Which one of the 196 Config:: systems should we use? Which database ORM should we use? Do we use mod_perl or FastCGI? Which email module is best for this project? How should we trap errors? Which MVC model should we use?

Certainly each option is a good solution for a specific kind of situation, but how do we know *which* module works for *this* solution?

Which toolset?

Do we use DBIx::Class + Catalyst, Class::DBI + HTML::Mason or straight DBI + CGI::Application? Should I write HTML widgets myself or use CGI, HTML::Widget, HTML::DOM or HTML::TurboForm? And above all, Which ones work together better?

I would rather have the options than not have the options, but it would be great if we could get some good documentation together that would detail how to build a web application using DBIx::Class, mod_perl and Catalyst, vs using HTML::Mason, FastCGI and Rose::DB::Object (just contrived examples of options, nothing more).

How to Scale Out?

If the web application is successful you will need to plan ahead for scaling. Scaling out without session-affinity (i.e. "shared-nothing") is arguably the more popular option because of cost (an additional web server is cheaper than an F5 Networks BigIP). However scaling out means sharing Session and Application state. It means managing more than one server. It means you need to be capable of deploying updates to all servers in a way that won't cause errors, undesirable behavior or downtime. In any case, it means much more than just setting up another server behind a load-balancer.

Development Sandboxes

While one programmer can handle dealing with the Dev->QA->Prod push just fine (in most cases), 2 developers or more can really cause problems. Since mod_perl generally means restarting Apache httpd each time a code change has been made, you may end up running a local version of the code (and possibly the database). Having a local version implies that you are using CVS, SVN or some other revision control system (yes, another set of options with no clearly-defined winner).

The total lack of a dead-simple IDE (like Visual Studio) means that we get different people using different tools. Again - another set of options with no clearly-defined winner (even though we all have our favorites). On the other hand, the ability to use whichever editor we want without relying on the IDE to generate code, line up dependencies or compile the project gives us the flexibility we have come to expect.

Switching from a Windows desktop to Linux (i.e. Ubuntu) saved me a great deal of pain, but it was a trade-off. Is there a Gnome equivalent to the super-simple Tortoise-SVN shell extension for Windows? Right-click->SVN->Update doesn't get any easier.

Which Dev—»QA—»Production Publishing Toolset?

Once you have made progress on your app and it's time to publish, you have more options:

• RCS Checkout on QA and Production
• FTP or rsync to QA and Production
• Make a PAR::Dist
• Use App::Build (though docs and examples need work).
• Use App::Install (though docs and examples need work).

Each of these options will need to account for configuration files and other differences between Dev, QA and Production.

Maintenance

Thus spake the master programmer:
``Though a program be but three lines long, someday it will have to be maintained.''

While writing a test suite for a single module is an exercise in quality and partial proof of the author's will to produce something of real value, we tend to think nothing of releasing a mission-critical web application without a single test. Not even one. Perhaps the reason is that until Catalyst or Apache2::ASP it was very difficult to write unit tests for web applications because, well, you had to execute them from within a webserver, and it was just a huge pain.

Having developed my last several projects on Apache2::ASP with full test suites built with Test::More et al, I can attest to the value of code-coverage (via Devel::Cover), profiling (via Devel::NYTProf) and regression-awareness with regard to web applications. Instead of wasting time clicking through the website (i.e. a 12-page loan application) entering every possible value, testing every possible input error, etc. I could just run the test suite in seconds or minutes and check to see if any of those bugs I was testing for had shown up.

If only I had web application test suites in the past. It would have saved countless hours and allowed me to add the features we really could have used, but didn't have time for. I could have done that refactoring without worry that application behavior had changed somehow.

Looking Ahead

The problems we face right now are Good Problems to Have. We have several options to choose from and there is a great deal of work happening in the areas I've pointed out above. I expect that the more people we have working on these problems, the more solutions we will end up with (yes, more options too).

With enough time and entropy the winners will reveal themselves. Until then, we can make our own choices.

[oknow]

Certainly each option is a good solution for a specific kind of situation, but how do we know *which* module works for *this* solution?

There does tend to be a huge variety of modules for every common task on CPAN, but I don't think that creates much of a problem in practice. If you ask for recommendations, or search for old ones, you're likely to find a small handful of commonly recommended modules for each common task.

It isn't like there are 196 advocates cheerleading for 196 different Config:: modules. I don't even think I can think of a half dozen config-related modules that I've seen people praise.

I would rather have the options than not have the options, but it would be great if we could get some good documentation together that would detail how to build a web application using DBIx::Class, mod_perl and Catalyst, vs using HTML::Mason, FastCGI and Rose::DB::Object (just contrived examples of options, nothing more).

You want a tutorial on how to "write a web application using <DB Abstraction Module>, <Templating System>, and <Runtime Backend>." There are likely a half dozen reasonable choices for the first two, and three or four choices for the backend.

I am not sure I understand why you need a tutorial explaining how to use three systems in concert when they are all so well isolated.

However scaling out means sharing Session and Application state. It means managing more than one server. It means you need to be capable of deploying updates to all servers in a way that won't cause errors, undesirable behavior or downtime. In any case, it means much more than just setting up another server behind a load-balancer.

It shouldn't mean much more than setting up another server. Just store your session information in your database and you'll be fine, all your front end servers will be able to hit it. This will work if you start with a combined database and web server, it will work when you split those out to their own boxes, and it will work as you add web servers. Scalability won't start getting expensive until you outgrow your database server.

Of course, that is pretty much how it is with any language, not just Perl.

While one programmer can handle dealing with the Dev->QA->Prod push just fine (in most cases), 2 developers or more can really cause problems. Since mod_perl generally means restarting Apache httpd each time a code change has been made, you may end up running a local version of the code (and possibly the database). Having a local version implies that you are using CSV, SVN or some other revision control system (yes, another set of options with no clearly-defined winner).

Again, these are the same problems you have with any language. You need whole other instances of all the services that your application needs. They may be another set of identical servers, a scaled down set of servers, or just might be running on someone's desktop (VMware and friends work great for this).

Thankfully, there is no clearly defined winner in the world of revision control. For the longest time, CVS was the only open source game in town. Now there is a ton of innovation and competition in this arena. You are doing yourself a serious disservice if you don't try one of the distributed systems, but that is a whole different story.

The total lack of a dead-simple IDE (like Visual Studio) means that we get different people using different tools. Again - another set of options with no clearly-defined winner (even though we all have our favorites). On the other hand, the ability to use whichever editor we want without relying on the IDE to generate code, line up dependencies or compile the project gives us the flexibility we have come to expect.

I won't touch a language that forces (or even very strongly encourages) me to give up my editor and use their IDE. It seems that with the MS languages I'd either be hobbled by their editor, or I'd be hobbled by mine. I suppose at least the choice would be mine, eh? :p

Each of these options will need to account for configuration files and other differences between Dev, QA and Production.

This isn't a Perl related problem, either. Every language has this problem.

The problems we face right now are Good Problems to Have. We have several options to choose from and there is a great deal of work happening in the areas I've pointed out above.

I don't think most would agree with you that points you bring up are actually problems, and more than half of your points are related to working with any language at all.

There are multiple solutions to most of what you see as a problem. The trouble is, most of them are already solved. The part you don't seem to like is that there is more than one solution for each of them.

There is more than one way to deploy a program to production, interact with a database, or scale a web site. The problem is that the right answer for you isn't the right answer for me. Maybe your application can scale well by doing a bunch of caching on the web servers and only keeping a single database server. Maybe that doesn't work well for me, and I need a database cluster. There is a reason these decisions aren't being made for you ahead of time.

With enough time and entropy the winners will reveal themselves. Until then, we can make our own choices.

Fortunately, there can be more than one winner. Most of the individual components you mentioned are very good at what they do. Thankfully, we can pick and choose the modules in each category that best fit our style and needs.

[dragonchild]

You want a tutorial on how to "write a web application using <DB Abstraction Module>, <Templating System>, and <Runtime Backend>." There are likely a half dozen reasonable choices for the first two, and three or four choices for the backend.

I am not sure I understand why you need a tutorial explaining how to use three systems in concert when they are all so well isolated.

Ah, but they aren't. For example, the support for using TT and DBIC with Catalyst is much stronger than using TT and DBIC with CgiApp. Or, using Text::Template and Rose::DB with Mason. And so on. While you can mix'n'match, there are some that work better with each other than with others. Hence, the usefulness of the requested documentation.

[talexb]

For example, the support for using TT and DBIC with Catalyst is much stronger than using TT and DBIC with CgiApp.

I disagree -- the module CGI::Application::Plugin::TT ties Template Toolkit and CGI::Application together very nicely, and you can see piles of plugins for CGI::Application if you do a CPAN search on 'CGI::Application::Plugin'. I used CGI::Application::Plugin::Session, CGI::Application::Plugin::DBH and CGI::Application::Plugin::Stream in a recent web application. I'm a fan.

[perrin]

the support for using TT and DBIC with Catalyst is much stronger than using TT and DBIC with CgiApp

I don't actually see much integration between Catalyst and TT or DBIC. It's not really possible when you take a hard line on being able to switch components. Jifty is the only really integrated perl dev framework I can think of.

[oknow]

Ah, but they aren't. For example, the support for using TT and DBIC with Catalyst is much stronger than using TT and DBIC with CgiApp. Or, using Text::Template and Rose::DB with Mason. And so on. While you can mix'n'match, there are some that work better with each other than with others. Hence, the usefulness of the requested documentation.

I suppose I should remember that it is rarely safe to speak in absolutes. The modules that play better with certain modules than others state that pretty clearly in their documentation, don't they?

Modules that don't play well with others are far less common than those that do.

[strat]

Hello,

in my current project (not a web app, system programming and automation), we use the following way to develop, test and do the roll out.

CVS: will soon be replaced by Subversion

UML: we chose the Rational ones from IBM, and TopCased as an Eclipse plug in for the developers who don't have licenses for the Rational stuff (only for documenting).

Eclipse (from YOXOS) with EPIC for Perl support, an integrated CVS (soon SVN) client (is as easy as tortoise), perltidy for formatting, and a lot of eclipse templates. It wasn't easy for me to switch from emacs (my favorite editor) to eclipse, but using the same platform independent tool for all developers is a big pro in my eyes. We just evaluate Jazz which looks very interesting.

Coding Standards: we use a subset of "Perl Best Practices" (those parts which make sense) and several enhancements

Perl and add ons: to get a standard environment, we use pkgsrc for Perl, libraries and some applications

Packaging: we use RPM for Linux, InstallP for AIX (and soon MSI for windows) and developed a little tool who does the packaging for us. The roll out is done in several stages, from develop (2 servers) to test (8 servers), to acceptance/integration test and then into production (about 200 server). We use a lot of automatic tests for step one and two.

Perl-Modules: lot's of them, but a list of them might not be helpful for you :-)

Bugzilla: for Bugtracking, and the eclipse plug in mylin to use most of its features from eclipse.

Update: I forgot bugzilla, and fixed some spelling errors...

[jdrago_999]

Thanks for sharing - it sounds like you and your team have got things worked out smoothly.

It would be great if you had the spare time to make a blog entry somewhere about the decisions that were made which led you to the mix of tools you have listed above.

[ysth]

With enough time and entropy the winners will reveal themselves.

I don't think so. I guess it depends on what you mean by "winners". I would call anything that manages to gather to itself a community of users and developers who can provide each other support and ideas, a "winner". But that applies to just about all the choices you mentioned.

[samtregar]

I argue that Perl suffers from "Too much of a good thing."

What evidence of this "suffering" do you see? Are people leaving Perl in favor of languages with fewer options, for that reason? Frankly, I would have left Perl long ago if not for CPAN, and I think CPAN doesn't "suffer" from this condition - it florishes because of it! Perl makes hard jobs easy because we all get together and put our best work on CPAN. If we decided we only wanted X config modules or Y template modules we'd run a very high chance of missing out on something great! Sure, there's a price in user confussion, but I think it's a very small price to pay for the gigantic boon of a healthy, vigorous development pool.

Certainly each option is a good solution for a specific kind of situation, but how do we know *which* module works for *this* solution?

Easily done. Just read Perlmonks - we're all more than happy to state our preferences when we're asking for help or answering questions. Of course that will never be enough to guide your choice alone - you'll have to do research yourself and take risks. Then report back and let us know how it went.

-sam

[jdrago_999]

While you and I might be very comfortable plucking the good stuff from CPAN (or rolling our own) there is a large number of would-be Perl users that are intimidated by the vast number of options.

They come from many different paths, but certainly one of the most common is the group that comes to Perl in need of server-side interactivity for their AJAX/Flash/Email form or similar.

Reading the Perl literature (and lore) you can see that we like to brag that Perl makes easy things easy, and hard things possible. We say things like "Perl lets you start out talking baby-speak." However in order to get up-and-running with something as simple as a JSON-aware guestbook backend for an AJAX website we have to understand many things:
• User permissions and paths for installing modules from CPAN.
• What CPAN is in the first place.
• Which JSON module do I choose?
• Which CGI modules do I choose? (Remember this is probably on a shared-hosting account.)
• What does "500 server error" mean?
Otherwise the would-be-Perl-user just runs off to $OtherLanguage and remembers Perl only as a difficult-to-use, impossible-to-understand language from the 1990's. Next time they want to do something interesting (i.e. image-thumbnails on the server, with filters or other effects) they will remember what a PITA Perl was that first time around and go with $OtherLanguage like last time. Even if $OtherLanguage doesn't have CPAN and the awesome library of tools, they somehow feel Perl is opaque, intimidating and out of reach.

When the person needs to pay rent (not "take risks") they will search Google and go with the first tutorial they find that works for them. Hand-holding is great when you are trying to learn (and pay your rent at the same time).

Although Ruby and PHP are technically inferior to Perl in several ways, they are very popular with newcomers because they are easy to get started with.

I think that if there were a kind of "Training Wheels Perl" environment that included the kinds of things people are doing with Ruby and PHP without too much pain, we would see Perl begin to gain in popularity with the newcomers. Perl could lose that "Difficult" image it has picked up.

I remember a while back, listening to a guy laugh out loud, "Oh no, we won't be using Perl for this (haha) no way. Those guys have got to have 10 pound brains just to understand it! No! (hahaha) We'll be using PHP. Perl is just way too complicated (hahaha)!" Perhaps a bit extreme, but that's the consensus.

Sure, maybe it's just marketing, but I know what I'm doing and it still takes me a few hours to go from a fresh install of CentOS/RHEL/Ubuntu/Fedora/whatever to a working server with the following installed:
• Apache httpd2
• Apache libapreq2
• mod_perl2
• MySQL 5
• DBI
• GD
• Imager
• DBD::mysql
• Apache2::ASP
• Class::DBI::Lite
• JSON::XS
• Test::More
• Devel::Cover
• Devel::NYTProf
I've streamlined the process a great deal and it still takes a while. I can imagine what it takes for a newcomer to get up-and-running with some of the more advanced tools like Catalyst + Template::Toolkit + DBIx::Class.

[BrowserUk]

something as simple as a JSON-aware guestbook backend for an AJAX website

I've been programming for a few decades, and Perling for 6 years, and that sentence means almost nothing to me. I'm vaguely aware that AJAX is roughly the web equivalent of "raw mode" for http, but I've not the vaguest clue what JSON is. And I've never wanted to write a guestbook. (Nor contribute to one!)

And I'll never understand the need of Perlers to want everyone else to use Perl?

I love my 20 year old Japanese sports car. It is economic to run, 40mpg when driven with a modicom of care, low service costs and barely a sign of rust after 20 years of non-garaged daily use. And yet has enough performance and handling to provide a few thrills on those rare occasions I get to take somewhere I can utilise it safely. But I never felt the need to encourage all my freinds to get one, never mind strangers. Indeed, part of the original reasoning for buying it was the small degree of exclusivity it provided.

If other people are happier to use $Otherlanguage, why does that bother you?

[mr_mischief]

"As simple as a JSON-aware guestbook backend for an AJAX website" is not a baby speak project. Besides what you list, people need to also understand: the stateless nature of HTTP, markup, JavaScript, the container the browsers provide for JavaScript, asynchronous server calls in JavaScript or some toolkit which abstracts them away, a database or filesystem storage solution for the data, what JSON is, How JSON can be useful, Perl, permissions and paths in general, CGI, some sort of security practices related to XSS, some security principles related to their backend data storage, and how to use an editor. There are plenty more I could list.

Getting started using baby speak means that to do something terribly simple, you don't need to specify a great deal of modules and type in a bunch of syntax. Compare Perl to COBOL or even Java. (The programs were found at the linked sites, and are their respective property. Used without explicit permission.):

print "Hello, World!";

000100 IDENTIFICATION DIVISION.
000200 PROGRAM-ID.     HELLOWORLD.
000300
000400*
000500 ENVIRONMENT DIVISION.
000600 CONFIGURATION SECTION.
000700 SOURCE-COMPUTER. RM-COBOL.
000800 OBJECT-COMPUTER. RM-COBOL.
000900
001000 DATA DIVISION.
001100 FILE SECTION.
001200
100000 PROCEDURE DIVISION.
100100
100200 MAIN-LOGIC SECTION.
100300 BEGIN.
100400     DISPLAY " " LINE 1 POSITION 1 ERASE EOS.
100500     DISPLAY "Hello world!" LINE 15 POSITION 10.
100600     STOP RUN.
100700 MAIN-LOGIC-EXIT.
100800     EXIT.

class myfirstjavaprog
{  
        public static void main(String args[])
        {
           System.out.println("Hello World!");
        }
}

There's simply no way to compare your idea of a "simple" programming task to that. Getting started in a language and writing one's first semi-serious application with bells, whistles, and buzzword compliance built in are two very different things.

[talexb]

Can you boil your position down to a single question? Even this post raises such diverse issues such as

• .. there is a large number of would-be Perl users that are intimidated by the vast number of (CPAN) options.
• .. in order to get up-and-running with something as simple as a JSON-aware guestbook backend for an AJAX website we have to understand many things:
• .. Ruby and PHP are technically inferior to Perl in several ways, they are very popular with newcomers because they are easy to get started with.
• "Training Wheels Perl"
• Sure, maybe it's just marketing, but I know what I'm doing and it still takes me a few hours to go from a fresh install of CentOS/RHEL/Ubuntu/Fedora/whatever to a working server with the following installed:
Some of these have nothing to do with Perl. Yes, there are many choices on CPAN. So?

And writing a JSON-aware guestbook probably requires you understand what a 500 Internal Server Error is in any web programming language, including PHP and Ruby.

Installing a new server with a bunch of packages takes a while, but are you doing this weekly? And if you are, why not standardize on a set of packages and go with that, updating your standard install quarterly?

I really don't see what problem you're trying to solve.

[JavaFan]

However in order to get up-and-running with something as simple as a JSON-aware guestbook backend for an AJAX website we have to understand many things:

Hmmm, if one finds doing a JSON-aware guestbook backend for an AJAX website "simple", I don't think such a person should have much trouble finding out what CPAN is, or setting up permissions of paths.

Note that user permissions have nothing to do with Perl, and neither do 500 server errors. But note that '500 Server Error' does have its own entry in perldiag.

Note further that CPAN isn't Perl. CPAN is a user network. Code on it doesn't have Larry permission, nor has it been blessed by the perl-porters. CPAN is not Perl. Repeat. CPAN is not Perl. Nothing can be derived from the existance of code on CPAN. Heck, there's not even a garantee that what is on CPAN is actual Perl code.

I think that if there were a kind of "Training Wheels Perl" environment that included the kinds of things people are doing with Ruby and PHP without too much pain, we would see Perl begin to gain in popularity with the newcomers. Perl could lose that "Difficult" image it has picked up.

So, your conclusion is that having a community that doesn't share code, or makes sharable code hard to find, is actually a good thing? Because it reduces options? Well, I guess it's always possible for a newbie block cpan.org on his/her firewall....

{Setting up a new server} I've streamlined the process a great deal and it still takes a while.

Then you haven't streamlined it enough. It's certainly not Perls task to streamline setting up your server. Most people will want to install many pieces of software that isn't written in Perl anyway. To be able to quickly install a new server, the correct answer is to use a repository, using whatever package manager is appropriate. I use yum and (Solaris) quickstart.

I can imagine what it takes for a newcomer to get up-and-running with some of the more advanced tools like Catalyst + Template + DBIx::Class

Well, they aren't "advanced" for no reason. But how would you solve it? Forbid the existence of Catalyst? Do you have any solutions?

[samtregar]

While you and I might be very comfortable plucking the good stuff from CPAN (or rolling our own) there is a large number of would-be Perl users that are intimidated by the vast number of options.

So what? Perl can't be all things to all people. I see absolutely no problem with sending people that can't deal with CPAN to Ruby or PHP. I'd certainly rather do that than lose anything from CPAN. All that diversity is what makes it go!

I think that if there were a kind of "Training Wheels Perl" environment that included the kinds of things people are doing with Ruby and PHP without too much pain, we would see Perl begin to gain in popularity with the newcomers. Perl could lose that "Difficult" image it has picked up.

I doubt it, but feel free to set one up and give it a try. Think of the fame and glory! You might want to re-name Perl while you're at it - the accumulated reputation isn't going to help you sell it as the new hotness. "Perl Lite" perhaps?

Sure, maybe it's just marketing, but I know what I'm doing and it still takes me a few hours to go from a fresh install of CentOS/RHEL/Ubuntu/Fedora/whatever to a working server with the following installed:

A few hours? What are you complaining about? You should try to get all that going with Ruby or PHP sometime! I think you'll be surprised by how hard it is.

-sam

[talexb]

I wish I had the time to go through and respond to each one of your questions, but it basically boils down to "use the appropriate tool at the appropriate time" and "pick one (or two) and go with that."

You close with the lines

With enough time and entropy the winners will reveal themselves. Until then, we can make our own choices.
Why do there have to be winners? Why not just agree that we've all got lots of tools in our toolboxes (and more each day), and it's up to our creativity to put them together in new and interesting ways.

[jdrago_999]

Why do there have to be winners?

Well, each option is a "winner" for some reason (otherwise it might not have been written).

Having options is great - having some way to qualify those options based on our current needs is even better.

If we have (just throwing numbers out there):
• 25 web frameworks
• 10 ORM layers
• 6 Charting/Graphing libraries
• 5 Email-sending modules
And I need to build a website that allows users to sign-up to receive daily sales data charts in graphic format, I have 25 * 10 * 6 * 5 options. That's 7500 possible options, and that kind of site is not exactly complicated.

I don't know about you, but I haven't tried all 7500 possible combinations to know which works best. What I think works fine could really stink. I could really be missing out on something great, and not even know it.

If we had some kind of objective means of qualifying the many modules and how well they work together, that would be great. I don't expect (or want) to see Perl end up with a J2EE/.Net FCL-style "Your Way Is Inevitable. You will be assimilated" One True Way of doing anything.

Having solved just about every problem there is with Perl, I think we should (as a community) be able to find some way to rate combinations of modules and techniques as they apply to the various problems out there.

Maybe a kind of "create-a-combination" tool that then allows you to vote on it. If someone else has already created a combination identical to yours, you just go to that combination instead. Similar combinations would be displayed near one another. It would lend context to the huge array of options we have, and allow the community to quickly ascertain the group opinion of a specific combination of modules. Things like "What worked?" "What didn't work?" "What issues arose?" and so on.

An idea anyways.

[talexb]

• 25 web frameworks
• 10 ORM layers
• 6 Charting/Graphing libraries
• 5 Email-sending modules

Sounds like you need to create a survey (Survey Monkey comes to mind) and ask people to take it. But why ask people for their opinions when you experience may not be the same as theirs?

I know I was keen to use an ORM interface to my database and spend a week trying to wrap my brain around Class::DBI. I hung out on IRC, asked all sorts of stupid questions and got nowhere, so decided to stick with just writing my own queries and commands instead of doing database stuff programmatically.

I also tried using CGI::Application about 6-7 years ago and couldn't figure it out, but gave it another shot two years back and it worked out great.

On the other hand, I've used Template Toolkit for half a dozen years and think it's fantastic. As long as it stays up to date, I won't need any other templating library.

But those are my experiences, and they reflect my own strengths and weaknesses. You probably have different experiences, so my choices won't necessarily match with yours. So even doing a survey of what everyone uses won't help you -- you need to figure out what works for you.

And don't think about needing to pick one of 7500 choices -- look at each piece individually. Being as this is Perl, it's pretty likely that your best choice for each part will work fine with your other choices.

Really, I think you're over-thinking this whole thing. Pick some tools that you're comfortable with and Just Do It.

[mr_mischief]

I don't know about you, but I haven't tried all 7500 possible combinations to know which works best. What I think works fine could really stink. I could really be missing out on something great, and not even know it.

You don't need to find 7500 ways that work. You need to find one way that works well enough. If it's fast enough to develop, easy enough to maintain, and scales well enough in production then your problem is solved.

One of the first things you learn in an economics class is the importance of opportunity costs. There is always a different way to spend your money or your time. The most important lesson about opportunity costs, though, is not that you must be perfectly methodical in your evaluation of them. The most important thing about opportunity costs is that if you spend all your time trying to figure out how to spend your time and money, then you'll have no time or money to spend and will never spend it on anything useful anyway.

Pick which trade-offs are acceptable to you and which ones are not. Then, be prepared to actually make a decision and follow it through. If you find you can't complete a task reasonably well with what you've chosen the first time, that's an expense. It's probably a smaller expense than never making the decision because you were still studying the options when it was time to act.

More information would be nice, and there have been numerous metadiscussions about the module selection process here already. However, this is not a perfect world and we often must act with incomplete or even inaccurate information. There's a large middle ground between acting too hastily and acting too cautiously. Try to find that middle ground, because that's where things get accomplished.

[oknow]

And I need to build a website that allows users to sign-up to receive daily sales data charts in graphic format, I have 25 * 10 * 6 * 5 options. That's 7500 possible options, and that kind of site is not exactly complicated.

It is more than a little disingenuous to say that you need to test anywhere near 7500 combinations of modules to find out which ones work well together. For the most part, all you have to do is pick the module in each of the four categories that works best for you.

How do you know which individual components to choose from the vast choices? I'd start with modules I've used in the past, asking friends and colleagues for advice, and then searching the internet/perlmonks for discussions on the same topic. If those options fail, then it is time to get my hands dirty and start trudging through CPAN. Thankfully, when there are 126 Config:: modules there tend to be a lot of people with opinions of a small subset of them.

Having solved just about every problem there is with Perl, I think we should (as a community) be able to find some way to rate combinations of modules and techniques as they apply to the various problems out there.

Have you actually had this problem a lot in practice? I don't think I have ever looked into using a module that didn't explicitly state in the documentation that it was made to operate with other particular modules.

This particular problem seems to be the exception and not the rule. I've never been bitten by it. I can't think of anyone I know telling me they'd been bitten by it, and I think you're the first person I have seen mention it at all.

Maybe a kind of "create-a-combination" tool that then allows you to vote on it. If someone else has already created a combination identical to yours, you just go to that combination instead.

You would end up with a vast number of positive votes in this system. You'd be better off having people submit just the combinations that are more difficult to use together. Otherwise we'd just end up with a bunch of noise saying Data::Dumper plays well with damn near everything :).

[phaylon]

I think you're tackling the problem from the wrong side. You seem to want "I need to write $project_site, what combination of @template_systems, @database_layers, @web_frameworks will do the job best?"

I think it is much better to start by analyzing your project, what it needs, where it needs to go now, and where in the future. If you want to write, for example, a bulletin board, most template systems will work fine for that. But if you know you want extensibility, lots of plugins, and much view functionality in the templates, you might want to start looking at Template-Toolkit rather than HTML-Template. If you want to build a bulletin board thats easy to customize for non-Coders, and don't need any fancy features in the templates itself, I'd go to HTML-Template.

That's why I am usually not interested in surveys or lists of modules people prefer in general. It all needs context. In my experience you can learn a lot by looking at successful projects, their dependencies, and how they use them. And most Perl projects I know are happy to answer "So, you guys used $foo for your project, I'm thinking about using it for $bar, so I'd like to hear how it worked out for you" questions.

[LesleyB]

I think the choice of modules and what to use when can only come from experience of those modules, followed perhaps by shared experience and being taught by others.

Recommendations help but experience teaches us the limitations of a module.

I don't think there is a single prescribed way to do anything in Perl. There are a number of ways each of which has its strengths and weaknesses. The inevitable TMTOWTDI applies.

An optimal selection may exist for a given problem, but it may not be optimal for another problem and it may not be optimal over all time as the original problem changes.

[stvn]

You forgot about all the Inline:: modules, given them your possibilities explode into the trillions!

One of the greatest things about Perl (IMO of course) is that we have so many possibilities. Take a look at the current state of the Ruby on Rails community to see how well "One True Way" got them. There is a serious amount of fragmentation going on, why, because your way is not always good for me, and my way is not always good for you.

The key is that knowledge is shared, even if the code is not. I think it is perfectly fine to reinvent the wheel if you have looked at and studied the other wheels out there and are trying hard to learn from and not make their same mistakes. I think the Perl community, being as tight as it is, does this pretty well (although sometimes rudely).

[Anonymous Monk]

Ultimately, I read your post and I merely think "problems? What problems?" Perl is doing just fine without a standard ORM or a standard IDE or a standard framework or anything. Just like every other really successful open platform, in fact -- look at C++, look at Java, look at Python, none of them have a single standard framework or a single standard IDE. The only cases where lockin is accompanied by any sort of success are certain languages peddled by a certain monopoly, which shall remain nameless for purely pragmatic reasons: I can't type their names because I can't find a hash key on this damn Mac keyboard. (Good thing I'm not writing any Perl, or it would be decidedly low on comments...)

And to pick up on a slightly irrelevant aside:

Is there a Gnome equivalent to the super-simple Tortoise-SVN shell extension for Windows? Right-click->SVN->Update doesn't get any easier.

Really? It sounds quite complicated to me. Certainly more effort than just typing "svn up". The command-line version also has the advantage of working everywhere, whereas if you insist on non-standard GUIs you'll be at a loss again when you find yourself working on a Mac one day...

[Jenda]

It would be "svn up filename" I guess. Or "svn up filename1 filename2 filename3 ....". Plus you have to learn the commands first. It's best to have both, GUI and command-line. And use each one when it makes sense.

[Anonymous Monk]

With enough time and entropy the winners will reveal themselves.

Ah, but if were only so easy to determine who the winners are. Even if we could determine what's most popular in terms of developer community, that would not reveal popularity in terms of real use, and that would not tell us if the developers and end users are happy with the winner.

In other words, poor solutions can persist.