Code Generation in Perl

Some time ago, I published an article about an open source Customer Relationship Management (CRM) application I was working on.

What I didn't tell you at the time, is that I was also working on a code generation tool which would allow me to generate a database-driven web application, just like this, in minutes.

The first version, called auto_coder, took an SQL script and generated all the required Perl scripts, modules and templates. The problem was that SQL isn't extensible: there's only so much information you can infer from a SQL script.

The solution was to use an extensible markup language -- XML for short :) -- to describe the database and all the required information.

This lead to a new version of my code generator, which I called "AppML" ("Application Markup Language").

Here's a small application described in AppML:

<project name="sample" title="Sample Application">
  <table name="contact" caption="contact" descriptor="name">
    <field name="name" type="varchar" size="40" notnull="1" caption="n
+ame"/>
    <field name="organization" type="varchar" size="40" caption="organ
+ization" visible="1"/>
    <field name="phone" type="varchar" size="40" caption="work phone"/
+>
    <field name="fax" type="varchar" size="40" caption="fax"/>
    <field name="mobile" type="varchar" size="40" caption="mobile"/>
    <field name="email" type="varchar" size="40" caption="e-mail" visi
+ble="1"/>
    <field name="notes" type="text" caption="notes"/>
  </table>
</project>
[download]

That's it!

Given the XML file, a Perl script will generate the complete Perl code, Class::DBI modules, templates (using Template Toolkit), documentation and diagrams automatically.

Of course, you can include as much details as needed. For instance, a project can have multiple tables and relationships:

<project name="crm" title="CRM">

  <table name="sector" caption="sector" descriptor="description">
    <field name="description" type="varchar" size="40" notnull="1" cap
+tion="sector"/>
    <field name="comments" type="text" caption="comments"/>
  </table>

  <table name="comp_size" caption="company size" descriptor="descripti
+on">
    <field name="description" type="varchar" size="40" notnull="1" cap
+tion="size"/>
    <field name="comments" type="text" caption="comments"/>
  </table>

  <table name="occupation" caption="occupation" descriptor="descriptio
+n">
    <field name="description" type="varchar" size="40" notnull="1" cap
+tion="occupation"/>
    <field name="comments" type="text" caption="comments"/>
  </table>

  <table name="contact" caption="contacts" descriptor="display_name">
    <rel name="sector" type="association" table="sector" caption="sect
+or"/>
    <rel name="comp_size" type="association" table="comp_size" caption
+="comp_size"/>
    <rel name="occupation" type="association" table="occupation" capti
+on="occupation"/>
    <field name="first_name" type="varchar" size="40" notnull="1" capt
+ion="first name" visible="1"/>
    <field name="last_name" type="varchar" size="40" notnull="1" capti
+on="last name" visible="1"/>
    <field name="display_name" type="varchar" size="40" notnull="1" ca
+ption="display name"/>
    <field name="organization" type="varchar" size="40" caption="organ
+ization"/>
    <field name="address" type="varchar" size="40" caption="address"/>
    <field name="city" type="varchar" size="40" caption="city"/>
    <field name="state" type="varchar" size="40" caption="state"/>
    <field name="country" type="varchar" size="40" caption="country"/>
    <field name="phone" type="varchar" size="40" caption="work phone"/
+>
    <field name="fax" type="varchar" size="40" caption="fax"/>
    <field name="mobile" type="varchar" size="40" caption="mobile"/>
    <field name="email" type="varchar" size="40" caption="e-mail"/>
    <field name="notes" type="text" caption="notes"/>
  </table>

</project>
[download]

I'm releasing AppML under the GNU General Public License, and the complete source code is available for download.

Any feedback will be appreciated!

Best wishes,

Nelson

--
Nelson Ferraz
GNU BIS

Comment on Code Generation in Perl Select or Download Code

Replies are listed 'Best First'.
Re: Code Generation in Perl by chromatic (Archbishop) on Jun 16, 2004 at 22:51 UTC
I like that idea, but I've never liked having to write XML by hand. What's the benefit over an API more like: `my $project = APPML::Project->new( name => "sample", title => "Sample Application" ); my $table = $project->add_table( name => "contact", caption => "contact", descriptor => "name" ); my $vc40notnull = AppML::FieldType->new( type => "varchar", size => "40", notnull => "1", ); $table->add_fields( [ name => "name", caption => "name", type => $vc40notnull, ], [ # more fields here... ] ); $project->create();` [download] That's the sort of interface I'd create. Maybe it's appropriate to make that write out the XML for you, but I'd personally skip the step of writing out XML, reading it in, and translating it into internal data structures.	[reply] [d/l]
Re^2: Code Generation in Perl by nferraz (Monk) on Jun 17, 2004 at 17:16 UTC
The short answer is: I didn't want to be tied to Perl; AppML can generate code in any programming language. (Although Perl is the first one.) Here's AppML output: nferraz@debian:~/appml$ ./appml.pl xml/sample.xml meta/db/create.tmpl -> sample/db/create.sql meta/db/view.tmpl -> sample/db/view.sql meta/db/stats.tmpl -> sample/db/stats.sql meta/db/drop.tmpl -> sample/db/drop.sql meta/pl/base.tmpl -> sample/lib/sample.pm meta/pl/base_cdbi.tmpl -> sample/lib/sample/cdbi.pm meta/pl/cdbi.tmpl -> sample/lib/sample/cdbi/contact.pm meta/templates/header.tmpl -> sample/templates/inc/header.tt2 meta/templates/menu.tmpl -> sample/templates/inc/menu.tt2 meta/templates/stats.tmpl -> sample/templates/inc/stats.tt2 meta/templates/footer.tmpl -> sample/templates/inc/footer.tt2 meta/templates/frm.tmpl -> sample/templates/contact/frm_contact.tt2 meta/templates/detail.tmpl -> sample/templates/contact/detail_contact. +tt2 meta/templates/list.tmpl -> sample/templates/contact/list_contact.tt2 meta/templates/tbl.tmpl -> sample/templates/contact/tbl_contact.tt2 meta/doc/install.tmpl -> sample/doc/INSTALL meta/doc/plan.tmpl -> sample/doc/sample.html meta/doc/dot.tmpl -> sample/doc/sample.dot meta/data/strings.tmpl -> sample/data/sample.txt [download] As you can see, the source metafiles are divided in directories: meta/db meta/pl meta/templates meta/doc meta/data These metafiles are nothing more than text templates, used to generate code; if I wanted to generate Java code, I would probably create a new directory to store Java metafiles.	[reply] [d/l]
Re: Code Generation in Perl by Zaxo (Archbishop) on Jun 16, 2004 at 21:01 UTC
In applm.pl, you say `$SIG{__WARN__} = sub { warn @_ unless $_[0] =~ /Use of uninitialized value/ };` Why not just this? `no warnings 'uninitialized';` Your program needs pod badly, there is no documentation. After Compline, Zaxo	[reply] [d/l] [select]
Re^2: Code Generation in Perl by nferraz (Monk) on Jun 17, 2004 at 17:43 UTC
Yup, I must improve documentation and code style. I just wanted to release the code as soon as possible! ("Release soon, release often", they say) Expect a revision in a few minutes.	[reply]
Re: Code Generation in Perl by perlfan (Vicar) on Jun 17, 2004 at 00:40 UTC
I think that is pretty bitchin'. All technical implementation critique's aside, it is at the very least extremely straight forward and useable for quick and dirty application. Good job, and thanks for posting it.	[reply]
Re^2: Code Generation in Perl by nferraz (Monk) on Jun 17, 2004 at 19:23 UTC
Thanks! I've successfully deployed some web applications using auto_coder and AppML. I'm sure it needs some tweaking, but I hope it will be useful to more people.	[reply]
Re: Code Generation in Perl by autarch (Hermit) on Jun 18, 2004 at 03:19 UTC
Have you looked at Alzabo at all?	[reply]
Re^2: Code Generation in Perl by nferraz (Monk) on Jun 18, 2004 at 13:19 UTC
If I understand it correctly, Alzabo is an "object-oriented wrapper"; applications created with Alzabo require Alzabo to be executed. AppML is a code generator; you describe what you want, and AppML will generate thousands of lines of Perl, SQL, and templates, to meet your project description. You can tweak the generated code, or, preferably, you can tweak the generator -- so the improvements will be applied to the next projects as well. Futhermore, AppML can generate code in other languages, and can use other open source platforms and frameworks like Alzabo or Maypole.	[reply]
Re: Code Generation in Perl by Jenda (Abbot) on Jun 19, 2004 at 10:23 UTC
If you needed a little more info that the SQL already contained you could have added it in there, eg. in the form of some specialy formated comments. That way your users do not have to learn a whole new "language", just a few additional attributes. And if someone does have the SQL ready, he/she doesn't have to rewrite it into XML to be able to use your generator. I would prefer `create table occupation /#descriptor="description"/ ( description varchar(40) not null /#caption="occupation"/, comments text )` [download] to `<table name="occupation" caption="occupation" descriptor="description" +> <field name="description" type="varchar" size="40" notnull="1" cap +tion="occu +pation"/> <field name="comments" type="text" caption="comments"/> </table>` [download] anytime. But that's maybe because I would prefer anything to XML. Jenda Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. -- Rick Osborne	[reply] [d/l] [select]
Re^2: Code Generation in Perl by nferraz (Monk) on Jun 19, 2004 at 13:19 UTC
Interestingly, that's exactly what I did in the first versions of auto_coder. One of the problems with this approach is that we don't have an easy way to define broader project properties -- we're too focused on the database design. Moreover, we must write different scripts for different databases; with XML we can write a single source and generate different scripts. Another advantage of XML is the avaibility of modules like XML::Simple, which translates a XML source into a Perl data structure like this: `$VAR1 = { 'table' => [ { 'descriptor' => 'name', 'caption' => 'contact', 'name' => 'contact', 'field' => [ { 'notnull' => '1', 'caption' => 'first name', 'name' => 'name', 'type' => 'varchar', 'size' => '40' }, { 'visible' => '1', 'caption' => 'organization', 'name' => 'organization', 'type' => 'varchar', 'size' => '40' }, # ...and so on.` [download] In other words, extending SQL with comments was my first approach, but it had its shortcomings.	[reply] [d/l]


laziness, impatience, and hubris
	PerlMonks