Follow up to RFC: Templating without a System

[shmem]

Dear Monks,

I don't know whether bringing up a topic twice in short time is a good idea, but since the rating of RFC: Templating without a System was 77% positive, I embolden myself to do so. Add to this the fact that the comments on it boiled down to either don't reinvent the wheel or no need for this, (with only one positive comment about the approach I took), and you may agree with me thinking that I haven't got much comments on the core of the topic, which is: make templating to be as simple as using format.

I'll address the objections first.

• no, I really had a strong need to do it.
• no, I don't reinvent the wheel, but I strip it from axes, brakes and hydraulic steering system
• and yes, I did it for reasons of logical dissent.

On the need:

I am just embarassed with modules meant to help in programming which force me to first give up, and then re-gain the flexibility and power that perl provides by studying carefully their interfaces and proceedings and following the ways others have pre-thought for me, those not being perlish ways but rather ways module authors contrived that can be done with perl.

On the wheel mantra, just to cite myself:

I don't reinvent the wheel, I just want a wheel with no car around, just in case I want a nifty chopper.

On the logical dissent:

If I use a module for a specific task (or a bundle of tasks which happen to be done on a specific well-defined chunk of "data", whatever that is), I expect it to do it's task and, other than that, stay out of my way.
I don't expect it to forbid me anything that perl allows; I don't expect it to impose me specific a-doings beyond what perl expects me to do, and I don't expect it to force me to write my code in a way that fits the way of thinking of the module's author.
For instance, OO programming is a tool, which you may or may not want to use. What templating stuff is left on CPAN, if you just happen not to follow the OO paradigm?

What is this templating stuff really about?

Templating...

... is the very task Larry Wall invented perl for in the first place, when he frowned upon sed and awk and decided to write his own tool to do reports. I think of templates as an enhanced format and of rendering as something like write.

As with perl's builtins format and write, template processing is:

• provide some preformatted stuff with placeholders
• interpolate some program's state into those placeholders
• output the result

That's it. No more about it. All extra's just feeping creaturism and is outside the task at hand.

But wait, you say, what about pipelining, callbacks, inclusion mechanisms, mini-language, namespace separation, lazy loading, caching and all other features we need of templating?

Well, these are all things that perl pretty well actually does, and writing our own mechanism for that is just overkill and re-inventing the wheel - with a wheel. Or wrapping syntactic sugar around perl's core features, for that matter.

To do that may seem plain silly. Others think it's The Right Thing To Do.

As for me, I don't care, but I want to achieve this task just with the perl core, using maybe some well-tested helper modules for specific tasks. Oh, wait... some? No, just File::Temp.

I'm so fed up with syntactic sugar that I'll get programmer's diabetes some day - I think there's enough of that sweet stuff the perl core:

• lazy loading is done via require at runtime (think POSIX)
• anonymous subs prevent namespace pollution
• require returns the last evaluated expression of a required file
• anonymous subs get garbage collected
• interpolation of variables during print is way faster than s///
• pipelining boils down to while(<>){ } and for(@ary){ }
• a callback is calling a sub from within a sub
• output can be delayed writing into another filehandle than STDOUT, or caching results in memory

This list of assertions is trivial (and incomplete). But if you work with these you get an enhanced format at no cost, and all the other stuff - pipelining, swapping output of $a with $b, caching etc - along with it.

I boldly state that all templating stuff at CPAN is over-featured and doesn't provide the basics of templating in a simple and portable way. Otherwise, why are there so many of them? why does each of them, beyond their basic task, provide extras like form validation, session management, URL escaping and so on? Why so much duplicated effort? Why do they need oodles of extensions and wrapper modules which provide access to other modules from their namespace and not the other way round?

After all, format and enhanced format - templating - are helper thingys, not application frameworks.

I line out my thoughts about how it could be done briefly in what follows.

Note: package Servlet renamed to OI, function servlet renamed to oi.

walkthrough

we have

• some program's internal state - variables
• a template with placeholders for that variables.

The task is to get those variables into the template. There may be others than just scalars to insert. Lists require loops. We might need conditionals because we want to use the same template for different purposes.

the provision...

...to insert something into some other content/context/whatever is - apart from perl builtins - a sub in perl, no matter if it's a function or a method. We'll use a sub, then, pass in data as usual and expect the sub to insert our data in the right place.

the template...

... happens to be none-perl around some perl (or metaperl, that is something which is later translated into perl) statements. The container is some other stuff than perl, that's why I say it's around. Well, you could also say that perl is inside the foreign stuff, but that doesn't help. I look at it from the perl point of view, not from the foreign stuff's point.

the provision...

...to output the filled template in perl is - boiled down - plain and simple:

        print;

or rather a bunch of print calls (yes, with control statements if necessary).

perl's view

Since we look at the template "with perl's eyes", let's just make clear it actually is perl. For that, we just turn it outside in. We make the surrounding stuff into print statements and leave the perl code as is. We wrap it all into a sub. Now the template is callable from perl.

Let's call the subroutine that does this mnemonically oi, for outside in. The subroutine oi returns the sub, which is the template, but now suitable for perl.

What's that returned subroutine's name? It has no name, so it's an anonymous sub.

to store or not to store

We could turn the template outside in every time we need it, but what for? It must be done only if the template itself - it's source - changes. So we write the converted template into a file. This file can be required at runtime. Now we did - by the way - provide lazy loading.

where shall we put it?

perl already has a convention for storage of lazy loaded functions, established with AutoSplit and AutoLoader. We stick to that and store the converted template in the directory auto/__PACKAGE__ (where __PACKAGE__ is the package of the caller). The root of this directory defaults to the current working directory, unless we say otherwise and push (or unshift) that root into @INC.

naming

Each template has a file name, so we just take that. Strip the extension (if any), append .al and done. Again, we go with what's already established in perl's core.

variables and scoping

We differentiate (as everywhere else) between lexical and package scoped variables. Global variables are globals, so no care about them, except the "usual suspects" - the global special variables, which will be localized inside the generated sub.

Suppose we have a template example.html used in package Foo, which uses the package local variables $pkg_var1 and $pkg_var2. In example.html we use the variables $foo and $bar, which are passed in as an argument list.

Calling oi

    package Foo;
    use OI qw(oi);
    use strict;
    our ($pkg_var1, $pkg_var2);

    $sub = oi(
        in  => 'example.html',
        my  => [ qw ($foo $bar)],
        our => [ qw ($pkg_var1 $pkg_var2 ) ],
    );

the resulting file auto/Foo/example.al contains

    package Foo;
    use strict;
    our ($pkg_var1,$pkg_var2);

    return sub {
        my ($foo, $bar) = @_;
        local ($_,$\,$/);

        # outside in code here

    };

calling the sub

After the file example.al is written, we can use it:

    package Foo;
    use OI qw(oi);
    use strict;
    our ($pkg_var1, $pkg_var2);

    $pkg_var1 = 'var 1';
    $pkg_var1 = 'var 2';
    
    my $sub = oi(in  => 'example.html')
    $sub->('page title', 'header line');

That's it, basically. Proceedings that pretty much adhere to perls core.

adding syntactic sugar

Oh no... again? C'mon, just 2 cubes.

first cube: Since we have files that pretty much look like AutoLoader files, we could use AutoLoader to pull them in. Alas, AutoLoader wants named subroutines in the *.al files. If AutoLoader would check if the value returned by require is an anonymous sub, we could use it. See this patch that would make AutoLoader goto &$subref. And we could, if use OI qw(AUTOLOAD) and telling oi to do so, provide for calling our converted templates by name:

    package Foo;
    use OI qw(AUTOLOAD oi);
    use strict;

    example('page title', 'header line');

second cube: We might not just have the output of our template printed, but have it return the content. But we already have made the foreign stuff into print statements. What can we do? We could setup a custom print which cares about that. But that would lead to evaluate a conditional in each call to, say, oi_print. Instead, we use File::Temp and write to a temporary file. We save the current filehandle, select our own, and print there, later restore the old filehandle. And, for the sake of simplicity, we make use of wantarray to determine whether we want our template 'just print', or return an array of lines.

Doing that, we can move the output around and reassemble it freely. If we are using templates stacked like a Matryoshka_doll, we can gather all output in just one tempfile.

extending

I've said no word so far about any markup of perl stuff inside templates. To make it easy for the parser of oi, the perl stuff must somehow be delimited. If the template happens to be HTML, I like using the HTML comment facility for that. So I place the perl code inside HTML comments.

But other ways could be (are) contrieved. If I'd rip out the code in TT which parses and converts the TT mini-language, and used that whilst parsing the template, I could as well insert TT tags, variables and control statements in the template's source and use them with oi.

One way would be writing packages like OI::HTMLComments, OI::TT and so on which provide the parsing stuff. I could then use templates written for any templating package in the same program, be they TT, HTML::Template, Mason, whatever.

And that is just what this is all about - a somehow generic approach to templating which imposes no artificial limits.

Rests to say, I want to make that stuff into a real module, complete with testsuite, README and all that's necessary, when it's ripe for that, and your comments are invaluable for this task. Where should I publish it for testing? Is appending to the existing nodes the right way?

The code in RFC: Templating without a System doesn't stand to scrutiny, and as for now nobody seems to have caught the bugs :-) I'll be cleaning that up. But the basic approach is visible therein.

Any comments on that approach? I'm looking for the best practice to do a simple task.

cheers,
--shmem Giordano Bruno

update: added a walkthrough