How do you organize your Perl applications?

[Wally Hartshorn]

Does anyone have any recommendations for how to organize your Perl applications? I'm not talking about the Perl code -- I'm talking about the files themselves.

Let me give an example. We are a very small shop, just 2 Perl programmers within a programming staff of about 10. Most of our time is actually spent on non-programming activities (HTML "coding"). Time actually spent doing Perl is sporadic, but eagerly anticipated. Most of it is CGI stuff.

In the Beginning: One program, One file

Initially, we followed the beginners' road of procedural programs, with print statements writing out HTML tags. (We were at least lucky enough to start of with CGI.pm, though.) That meant any given program consisted of a single *.pl file, which was plopped down into the cgi-bin directory.

Enter the HTML template files

Later, we discovered the joys of HTML::Template. And there was much rejoicing! No more print statements banging out HTML tags. Life was good -- and slightly more complicated, because now, in addition to the *.pl file, we had some *.html files that contained the templates. No problem, we just created directories in cgi-bin for each program and placed the *.pl file and the accompanying *.html files in the appropriate subdirectories in cgi-bin.

And the config files

Then we discovered Config::IniFiles. Ah! No more editing of the *.pl file just to change someone's email address. Just create a *.ini file and put that in the directory with the *.pl file and the *.html files.

And the modules

Then we wrote our first module, to be used by multiple programs to do error reporting. Okay... well, that doesn't go in the cgi-bin directory, that gets installed as a Perl module. Okay, we'll call it something like OUR::Errors.

Now we've written our first program to use CGI::Application. Using that, you create just a very small *.pl file, with the bulk of the code in a module. Okay... well, then that means that the *.pl file, the *.html files, and the *.ini file go over in the cgi-bin directory, but the bulk of the actual code -- the module itself -- gets stored with the other Perl modules.

And the unit tests

Oh, and we've also taken our first stab at doing automated testing, so we've also got a test.pl file that should go... uh... somewhere.

Now store it all in CVS ... somewhere

Added to the mix is trying to figure out how to best use CVS to work with all of this. Previously, when we wanted to work on "project-x", we would just check out "project-x" and have all of the files there for us. After making the needed changes, implementing the updated program was a simple matter of copying our working directory into the appropriate cgi-bin directory. (Like I said, we're a very small shop.)

Now, however, things are going two different places, and I'm not sure how they should be organized for CVS.

CPAN to the rescue?

I have a feeling that ExtUtils::MakeMaker or Module::Build would somehow be a part of the solution. I've experimented with ExtUtils::MakeMaker for a while, and have figured out how to get it to "make test" and "make install". However, "make install" only installs the *.pm file -- it doesn't know to install the *.html, *.ini, and *.pl files into the appropriate cgi-bin directory, and the Makefile looks far too intimidating for me to figure out.

In addition, when I do a "cvs commit", CVS commits a whole bunch of directories in 'blib' that have no known function to me, as well as some files (e.g. 'pm_to_blib') that seem equally mysterious.

Mission: Incomprehensible

Most of the stuff that I've read about this generally talks just about Perl modules and getting them ready for CPAN. I haven't really seen anything that talks about all of the other stuff. Is there something obvious I'm missing? Do I need to RTFM a bit more on ExtUtils::MakeMaker or Module::Build?

So, given these files:

• myprogram.pl
• myprogram.ini
• MyModule.pm
• mytest.pl
• mytemplate.html

How would you organize this stuff? Should it all be in one CVS project, or should there be multiple CVS projects? Should most of it go in a subdirectory within cgi-bin? Or with the module? Or somewhere else entirely? When I do a "make install", how can I get everything to go where it should?

With just the 2 of us, the issue isn't critical at this point. However, we would like to "do the right thing" and make sure that we don't leave behind a maze of twisty little Perl files, all different, for those who will eventually follow us.

Wally Hartshorn

(Plug: Visit JavaJunkies, PerlMonks for Java)

Edited 2003-04-04 by Ovid

[Ovid]

Here's roughly how we would do it at my work (assumes that this is under some site specific directory):

+---data/
|   myprogram.ini
|
+---lib/
|   MyModule.pm
|
+---t/
|   mytest.pl
|
+---templates/
|   mytemplate.html
|
+--www/
     |
     +---cgi-bin/

The main consideration is to ensure that nothing goes under the www/ folder unless you really need direct Web access to it. Otherwise, one slip in your configuration could give away information that you may not want others to see.

Cheers,
Ovid

New address of my CGI Course.
Silence is Evil (feel free to copy and distribute widely - note copyright text)

[PotPieMan]

I agree with your directory structure, but I would like to make one further recommendation: Instead of a data directory for configuration files, use the standard UNIX etc directory. This allows you to use data for actual data, such as comma-separated value files that a client sends you.

From the data directory, I generally move to a ddl directory for database definition files and a sql directory for SQL scripts.

--PotPieMan

[DapperDan]

Forgive me if this sounds pedantic, but how do you distinguish between DDL and SQL scripts? I only ask because I use a single directory (sql, although I like ddl as well) for this. Aren't they the same thing?

I also wonder if you have a sibling dml directory which has the definitions for a SQL phrasebook. To be honest, that idea only just occurred to me, but it does seems attractive. Something like:

+--sql
   +--dml  # database setup scripts and definitions here.
   +--ddl  # data files with phrasebook definitions here.

[Anonymous Monk]

I disagree with that recommendation.

There are many cases where you want to allow multiple configurations to co-exist on the same machine. For instance multiple developers on one machine, each with their own home directory. Or staging and production versions of the same site on the same machine, served from different IP addresses.

Of course if this is for broader distribution and not just for internal use, then be nice to administrators and put configuration files where the sysadmin will expect it - namely /etc. But for internal use...?

[perrin]

Ovid's directory structure is about the same as what I use. I would add a few specific things:
• One CVS project to rule them all.
• There's no need for MakeMaker or Module::Build if you are not creating packages for people to install on their own machines with an uncertain directory structure. A tarball is probably fine for your internal use.
• I don't put my own modules in the same place that Perl's library and site_perl stuff goes, because that stuff is owned by root. I put mine in a separate place and add it to @INC.
• Don't put ANYTHING that is not an executable or static file under your web server's docroot.
• If you use mod_perl, there's no need to put anything at all under cgi-bin. I just make PerlHandlers and map URLs to them in httpd.conf.

[waswas-fng]

One CVS project to rule them all.

Why? I think that would totally depend on how tightly these projects fit in the same scope? Can you say that is the best plan if these perl projects are lets say cgi point projects for different clients? There is a reason CVS supports multiple project and many good reasons to use them.

-Waswas

[perrin]

The description certainly made it sound like this was a single website. A single website would most likely be best served as a single CVS project. It would only make sense to split it up if there are truly multiple independent projects with different destinations.

[simon.proctor]

Here's mine:

docroot
logs
admin
templates
config
logs
Readme.txt
lib <---- may be called something else see below

It should be fairly explanatory other than the admin programs for any web application go in the admin folder and have a separate (sub)domain/virtual host. Naturally the docroot folder (document root) is for the web application main. The logs folder is both for any debug or normal running logs as well as any logs from the webserver.

If this isn't a web application, then the docroot goes but the admin folder normally remains (it depends). As for the lib folder. This holds any modules that I write that I don't find on CPAN or need to get from CPAN but cannot install and have to do a 'use lib' for.

One thing I always do is add a --help option to any command line scripts . I always forget how to use my own programs!

In the context of your files, ini goes in the config folder, html in the templates folder (or sub folder), module in the lib folder and as this isn't a web app, the perl script goes in the root. As you would expect, tests go into a 't' folder.

The readme is to describe where everything is and what it does. Possibly not Perl style but some of my command line apps get used by people not used to these things but who can follow instructions.

As an aside, when using templates I tend to create template aliases (see below). This allows me to change where files are in one place without hunting through code.

$templates->{'login screen'} = '/www/templates/login/form.phtml';

I also call all my files with the same endings. So config files end in .config, html templates end in .phtml and so on.

HTH

SP

[DapperDan]

As an aside, when using templates I tend to create template aliases (see below). This allows me to change where files are in one place without hunting through code.

$templates->{'login screen'} = '/www/templates/login/form.phtml';

Ahhh! A phrasebook for templates? That's a great idea.

[demerphq]

My team has had similar issues to which you describe. No CGI or html, but aside from that broadly similar. We have tended to go with a mixture of source control (source safe in our example) and CPAN packages. We tend to build a "project" in source safe that reflects our overall setup. Module bundles that make sense to be shared between projects are checked into projects with names like "lib-foo".

Now here is something to keep in mind, the part of CPAN package that you need is only the base stuff generated by h2xs (or equivelent) and any modifications and additions YOU make. Pretty much everything that gets left behind after you do a "make realclean". You dont need to check in anything that is generated by make/make test. Thus all the mysterious files and dirs you mentioned.

When we go to set up a project the first thing we do is install any "normal" CPAN bundles using CPAN as normal. Then we "get latest version" on the relevent unpacked private bundles held in CPAN. And run the install process by hand (make/make test/make install). Any modules go into bundles like this, ready for install later. All the rest of the project is then checked into sourcessafe "inplace". If you build your system appropriatly (and configuration system) approriately (we use a tweaked and extended Config::Inifile^*) a single change in the config file can handle a total relocation of the project.
*A useful addition we made was to allow for one config file in source control to be used on multiple machines of different configuration. This way key sections of the inifile can be made machine specific. This facilitates several developers working on different parts of the same project (with a shared INI file) on machines of different configuration. New stuff for the ini can be safely added to the common config file and checked out onto the various machines without worrying that doing so will screw up a given developers set up. And when changes need to made to these sections they are all in one place thus reducing the possibility of omission or other errors. Unfortunately the developers of the module rejected the patches we provided to them, so if this sounds useful you may need to reinvent the wheel. I could provide a patch I suppose. /msg me if you want.

For us this generally provides a flexible way to use source code control for both installation (installs generally amount to getting certain versions out of source safe onto the target enviornment, running the make process a few times and tweaking a few things), everything is under source control, and what makes sense to share is shared.

I delibertely wrote this before I read the other replys. Reason being that we currently have been doing cleanup and work in this very area (handling installs and source control) so its been on my mind lately. Thanks for asking the question I imagine that there will be useful advice from other people.


---
demerphq

_{<Elian> And I do take a kind of perverse pleasure in having an OO assembly language...}

[mandog]

Our structure is like that of other people. One somewhat useful thing we do is to put a ./install script in each CVS project directory.

Most of our ./install scripts are made of 'cp' commands. As simple minded as this seems, we've found it more effective than a written .txt checklist or our memory.

Our deploy sequence is something like:

 
./test.pl  
cvs commit
./install



email: mandog

[DapperDan]

This is partially rehashing some comments already mentioned above, but I want to emphasise them. As always, I welcome comments or suggestions from more experienced Monks if they are inaccurate.

In Apache:

1. You should NOT have code (scripts or modules) in a DocumentRoot or Alias directory. This is typically done using ScriptHandler with a file extension (e.g. .cgi or .pl). This will lead to your code being echo'd to the browser if the ScriptHandler breaks for whatever reason.
2. If you are using a cgi-bin directory (and this is to be preferred over using ScriptHandler), do not leave your modules, templates, tests, or other runtime 'assets' in this directory.

   Put them outside your cgi-bin, and (for perl code) use the lib pragma or @INC instead. You can do this easily with SetEnv PERL5LIB directory.

My tree has much in common with others already cited (and is revision controlled using svn 0.20), but the following directories are relevant:

+--htdocs  # Use with Apache <tt>Alias</tt> directive.
+--cgi-bin  # Use with Apache <tt>ScriptAlias</tt> directive.
+--lib  # Put this in you PERL5LIB environment variable.
+--templates
+--t

[Anonymous Monk]

A little off-topic, but since there is an Inline::C :), how do you organize your files in a c application? I've seen some that are just a mess of almost all header files and others that have a corresponding .c file for each one and limit the header files to prototypes, data structure definitions, and compiler statements. Could someone point me to a good resource on the subject? Thanks (:

[maksl]

php peoples like to put many files (conf, template, ..) in some non public dir often called inc or includes .. seems so (often seen on various webserver ;)
personally i like to use:


do conf;
use vars qw( $bla $whatever );

this config file is often in the same dir then the script itself. when executed, it doesn't return anything. all that fits in one cvs project.

[DapperDan]

Those interested in this thread may also be interested in this one off the mod_perl list:

(OTish) Version Control

[Oberon]

Since you've already got lots of great answers on the directory structure side of things, I thought I'd offer my opinions on the issue of CVS. As to (CVS) modules, what we use here is one (CVS) module per project, plus one (CVS) module for the common (Perl) modules.

Personally, I think using MakeMaker for this type of thing is overkill. You don't have to worry about possible installation of your modules on hundreds of obscure flavors of Unix-oid OSes ... just on one machine, and you know that one pretty well. I would go with mandog's suggestion of each project having its own install script. The way we do it here at my company is to have a single script called "rel" to release things, but of course it has to have a buttload of configuration files to tell it where things go and whatnot. Personally, I'm starting to think we should have done it the way mandog suggests too.

HTH.

[Wally Hartshorn]

Doh! Forgot to include a readmore tag. Could someone add one, please?

(Bad monk! No cookie!)

Wally Hartshorn

(Plug: Visit JavaJunkies, PerlMonks for Java)