Follow up to RFC: Templating without a System

[shmem]

Dear Monks,

I don't know whether bringing up a topic twice in short time is a good idea, but since the rating of RFC: Templating without a System was 77% positive, I embolden myself to do so. Add to this the fact that the comments on it boiled down to either don't reinvent the wheel or no need for this, (with only one positive comment about the approach I took), and you may agree with me thinking that I haven't got much comments on the core of the topic, which is: make templating to be as simple as using format.

I'll address the objections first.

• no, I really had a strong need to do it.
• no, I don't reinvent the wheel, but I strip it from axes, brakes and hydraulic steering system
• and yes, I did it for reasons of logical dissent.

On the need:

I am just embarassed with modules meant to help in programming which force me to first give up, and then re-gain the flexibility and power that perl provides by studying carefully their interfaces and proceedings and following the ways others have pre-thought for me, those not being perlish ways but rather ways module authors contrived that can be done with perl.

On the wheel mantra, just to cite myself:

I don't reinvent the wheel, I just want a wheel with no car around, just in case I want a nifty chopper.

On the logical dissent:

If I use a module for a specific task (or a bundle of tasks which happen to be done on a specific well-defined chunk of "data", whatever that is), I expect it to do it's task and, other than that, stay out of my way.
I don't expect it to forbid me anything that perl allows; I don't expect it to impose me specific a-doings beyond what perl expects me to do, and I don't expect it to force me to write my code in a way that fits the way of thinking of the module's author.
For instance, OO programming is a tool, which you may or may not want to use. What templating stuff is left on CPAN, if you just happen not to follow the OO paradigm?

What is this templating stuff really about?

Templating...

... is the very task Larry Wall invented perl for in the first place, when he frowned upon sed and awk and decided to write his own tool to do reports. I think of templates as an enhanced format and of rendering as something like write.

As with perl's builtins format and write, template processing is:

• provide some preformatted stuff with placeholders
• interpolate some program's state into those placeholders
• output the result

That's it. No more about it. All extra's just feeping creaturism and is outside the task at hand.

But wait, you say, what about pipelining, callbacks, inclusion mechanisms, mini-language, namespace separation, lazy loading, caching and all other features we need of templating?

Well, these are all things that perl pretty well actually does, and writing our own mechanism for that is just overkill and re-inventing the wheel - with a wheel. Or wrapping syntactic sugar around perl's core features, for that matter.

To do that may seem plain silly. Others think it's The Right Thing To Do.

As for me, I don't care, but I want to achieve this task just with the perl core, using maybe some well-tested helper modules for specific tasks. Oh, wait... some? No, just File::Temp.

I'm so fed up with syntactic sugar that I'll get programmer's diabetes some day - I think there's enough of that sweet stuff the perl core:

• lazy loading is done via require at runtime (think POSIX)
• anonymous subs prevent namespace pollution
• require returns the last evaluated expression of a required file
• anonymous subs get garbage collected
• interpolation of variables during print is way faster than s///
• pipelining boils down to while(<>){ } and for(@ary){ }
• a callback is calling a sub from within a sub
• output can be delayed writing into another filehandle than STDOUT, or caching results in memory

This list of assertions is trivial (and incomplete). But if you work with these you get an enhanced format at no cost, and all the other stuff - pipelining, swapping output of $a with $b, caching etc - along with it.

I boldly state that all templating stuff at CPAN is over-featured and doesn't provide the basics of templating in a simple and portable way. Otherwise, why are there so many of them? why does each of them, beyond their basic task, provide extras like form validation, session management, URL escaping and so on? Why so much duplicated effort? Why do they need oodles of extensions and wrapper modules which provide access to other modules from their namespace and not the other way round?

After all, format and enhanced format - templating - are helper thingys, not application frameworks.

I line out my thoughts about how it could be done briefly in what follows.

Note: package Servlet renamed to OI, function servlet renamed to oi.

walkthrough

we have

• some program's internal state - variables
• a template with placeholders for that variables.

The task is to get those variables into the template. There may be others than just scalars to insert. Lists require loops. We might need conditionals because we want to use the same template for different purposes.

the provision...

...to insert something into some other content/context/whatever is - apart from perl builtins - a sub in perl, no matter if it's a function or a method. We'll use a sub, then, pass in data as usual and expect the sub to insert our data in the right place.

the template...

... happens to be none-perl around some perl (or metaperl, that is something which is later translated into perl) statements. The container is some other stuff than perl, that's why I say it's around. Well, you could also say that perl is inside the foreign stuff, but that doesn't help. I look at it from the perl point of view, not from the foreign stuff's point.

the provision...

...to output the filled template in perl is - boiled down - plain and simple:

        print;

or rather a bunch of print calls (yes, with control statements if necessary).

perl's view

Since we look at the template "with perl's eyes", let's just make clear it actually is perl. For that, we just turn it outside in. We make the surrounding stuff into print statements and leave the perl code as is. We wrap it all into a sub. Now the template is callable from perl.

Let's call the subroutine that does this mnemonically oi, for outside in. The subroutine oi returns the sub, which is the template, but now suitable for perl.

What's that returned subroutine's name? It has no name, so it's an anonymous sub.

to store or not to store

We could turn the template outside in every time we need it, but what for? It must be done only if the template itself - it's source - changes. So we write the converted template into a file. This file can be required at runtime. Now we did - by the way - provide lazy loading.

where shall we put it?

perl already has a convention for storage of lazy loaded functions, established with AutoSplit and AutoLoader. We stick to that and store the converted template in the directory auto/__PACKAGE__ (where __PACKAGE__ is the package of the caller). The root of this directory defaults to the current working directory, unless we say otherwise and push (or unshift) that root into @INC.

naming

Each template has a file name, so we just take that. Strip the extension (if any), append .al and done. Again, we go with what's already established in perl's core.

variables and scoping

We differentiate (as everywhere else) between lexical and package scoped variables. Global variables are globals, so no care about them, except the "usual suspects" - the global special variables, which will be localized inside the generated sub.

Suppose we have a template example.html used in package Foo, which uses the package local variables $pkg_var1 and $pkg_var2. In example.html we use the variables $foo and $bar, which are passed in as an argument list.

Calling oi

    package Foo;
    use OI qw(oi);
    use strict;
    our ($pkg_var1, $pkg_var2);

    $sub = oi(
        in  => 'example.html',
        my  => [ qw ($foo $bar)],
        our => [ qw ($pkg_var1 $pkg_var2 ) ],
    );

the resulting file auto/Foo/example.al contains

    package Foo;
    use strict;
    our ($pkg_var1,$pkg_var2);

    return sub {
        my ($foo, $bar) = @_;
        local ($_,$\,$/);

        # outside in code here

    };

calling the sub

After the file example.al is written, we can use it:

    package Foo;
    use OI qw(oi);
    use strict;
    our ($pkg_var1, $pkg_var2);

    $pkg_var1 = 'var 1';
    $pkg_var1 = 'var 2';
    
    my $sub = oi(in  => 'example.html')
    $sub->('page title', 'header line');

That's it, basically. Proceedings that pretty much adhere to perls core.

adding syntactic sugar

Oh no... again? C'mon, just 2 cubes.

first cube: Since we have files that pretty much look like AutoLoader files, we could use AutoLoader to pull them in. Alas, AutoLoader wants named subroutines in the *.al files. If AutoLoader would check if the value returned by require is an anonymous sub, we could use it. See this patch that would make AutoLoader goto &$subref. And we could, if use OI qw(AUTOLOAD) and telling oi to do so, provide for calling our converted templates by name:

    package Foo;
    use OI qw(AUTOLOAD oi);
    use strict;

    example('page title', 'header line');

second cube: We might not just have the output of our template printed, but have it return the content. But we already have made the foreign stuff into print statements. What can we do? We could setup a custom print which cares about that. But that would lead to evaluate a conditional in each call to, say, oi_print. Instead, we use File::Temp and write to a temporary file. We save the current filehandle, select our own, and print there, later restore the old filehandle. And, for the sake of simplicity, we make use of wantarray to determine whether we want our template 'just print', or return an array of lines.

Doing that, we can move the output around and reassemble it freely. If we are using templates stacked like a Matryoshka_doll, we can gather all output in just one tempfile.

extending

I've said no word so far about any markup of perl stuff inside templates. To make it easy for the parser of oi, the perl stuff must somehow be delimited. If the template happens to be HTML, I like using the HTML comment facility for that. So I place the perl code inside HTML comments.

But other ways could be (are) contrieved. If I'd rip out the code in TT which parses and converts the TT mini-language, and used that whilst parsing the template, I could as well insert TT tags, variables and control statements in the template's source and use them with oi.

One way would be writing packages like OI::HTMLComments, OI::TT and so on which provide the parsing stuff. I could then use templates written for any templating package in the same program, be they TT, HTML::Template, Mason, whatever.

And that is just what this is all about - a somehow generic approach to templating which imposes no artificial limits.

Rests to say, I want to make that stuff into a real module, complete with testsuite, README and all that's necessary, when it's ripe for that, and your comments are invaluable for this task. Where should I publish it for testing? Is appending to the existing nodes the right way?

The code in RFC: Templating without a System doesn't stand to scrutiny, and as for now nobody seems to have caught the bugs :-) I'll be cleaning that up. But the basic approach is visible therein.

Any comments on that approach? I'm looking for the best practice to do a simple task.

cheers,
--shmem Giordano Bruno

update: added a walkthrough

[chromatic]

I found a great recipe for chocolate cake, so I went to the store and bought a bag of flour (I don't see why I should get my dishes dirty and baking makes my kitchen so hot in the summer) and ate it and it was absolutely the worst chocolate cake I have ever had.

(You absolutely do not have to justify code you write to anyone but your stakeholders, but the reason several experienced people said "Hmm... wait a minute, did you think about ____?" is either because they have done the same thing and it didn't work out for them or because you're so much smarter or luckier than all of them that they just don't understand what you're trying to do.)

[Ieronim]

The thing you have done is one more home-grown templating system. I won't say is it good or bad for YOU. It is your own code and if it fits your needs, it is generally good, because you get your job done.

But if your code will be re-used in future, its modificatition can become a horrible puzzle.

I have worked with a templating system I inherited from a programmer who stopped working on the project before I joined it. It seemed to suite the project well, but in a month I realized that new bugs appeared every time I tried to modify a bit in the templae or related code. I gave up and moved to HTML::Template, which is IMHO the best example of "Templating without a System" already availible in CPAN.

I work in pair with a web designer who creates HTML pages for the project and it was quite simple to teach him the new syntax since I allowed him to use %FOO% variable placeholders instead of messy <TMPL_VAR name="FOO"> ones (HTML::Template has a built-in ability to do such things).

Uph, it's a too long text for my broken English but I hope you understood me. I don't want to say that your idea is bad. But it can BECOME bad in future, if somebody tries to use it for his own job after you stop maintain it :)

[shmem]

I would like if you provided me some clue as to how home-grown and not home-grown modules differ.

Didn't HTML::Template start as something home-grown in the first place? Or was it the perl community that requested the author to write it?

Or do you differ by perl professionals that write such in their shop compared with random perl hackers writing stuff?

...after you stop maintain it

Erm, first I've got to finish it and see if it makes it's way to CPAN - which I doubt, as for me; there are many people out there that are better in writing such a thing as I myself. But I think it should be done.

--shmem

[Ieronim]

The only criterion to call the module "not home-grown" is the number of people have chosen it :)

I'll try to explain my idea.
Of course modules (in general) are written by random perl hackers ;) And at the start of the way all CPAN modules are home-grown — and your module too, if it becomes availible to CPAN. But then modules that do popular tasks become popular. I'll call it the community proof. When many peole use the code, they definitely find some odds and bugs. Then the author of the module fixes it — or not. In the second case the usage of the module usually stops.

But if everything is OK, after some cycles of bug finding and fixing the module becomes community-grown. And I prefer community-grown modules to home-grown, of course :)

I think it's obvious :)

[tye]

require returns the last evaluated expression of a required file

No, that is only true the /first/ time that a file is required. I agree that it would be very nice if Perl allowed require (and use) to return something useful (in particular, a 'factory' that defaults to just the package name). But that isn't the case yet, even though it may appear that way if you do incomplete testing.

[shmem]

That is no great deal since the requiring is done only once. The package holds a lexically scoped hash of anonymous subs keyed on the filename. On a second call of servlet with a specific filename that sub is returned and can be used; the generated whatever.al must not be required twice - except on code change of the source. That is what the timestamp checking is about: if the source file is newer, the file is deleted from %INC and is thus required again. The new sub overwrites the old one. In short, this happens:

package Foo;
$f = 'testsub.pl';
$s = eval "require $f";
delete $INC{$f};
$t = eval "require $f";
print "$_\n" for ($s,$t);
__END__
CODE(0x8167948)
CODE(0x81679a8)

And yes, I have to write a testsuite. (is there a leak?)
And yes, having a synopsis to talk about would be nice.


I'll update the code in the previous post sometime really soon (today or tomorrow).

--shmem

[tye]

So, not only are you using an undocumented feature of require, but you are also using it in a manner contrary to its purpose. In a code review, I'd reject such use unless you documented and enforced your unusual requirements. That is, you should check %INC before requireing so you can report that the file got required already (and include a comment about why that matters and that the feature of require that you are depending on is not documented).

Even better, Perl already has a tool that does what you are using require for: do $file. Use that instead.

[theorbtwo]

After reading the beginning of your missive, I wonder if you've considered Text::Template.

[shmem]

Yes, I did. The big difference between Text::Template and mine is the outside-in approach and the conversion of a template into an anonymous sub, apart from simplicity to have it as generic as possible. But Text::Template (as well as all other packages I've seen so far) are a great source for studying, to get me think of pitfalls I might not yet have considered.

Just two details of Text::Template to highlight my concern: the usage of Safe in Text::Template, and it's concern about tainted data. Well, I think that's none of the templating functions's business, but of the caller.

The builtin parser in Servlet (or OI) is for HTML, the plain text extension would be a bit more than just three s///, because it has do deal with whitespace properly.

Point is, I want something as close as possible to format and write; and with effective caching.

--shmem

[spiritway]

--shmem Giordano Bruno

Hmm... Bruno was burned at the stake for heresy... are you feeling somewhat martyred?

[shmem]

Uh, no, not at all. Being burned at stake was nothing of Bruno's business. But sometimes I feel like being heretic... ;-)

--shmem

[TGI]

FWIW, I find your critique interesting and look forward to seeing your ideas develop.

Your detractors are correct about their key points:

• CPAN is cluttered with templating systems.
• Writing one's own (bad) templating module seems to be a standard part of a Perl programmer's development. (I wrote one, it sucked. When I undestood CPAN I ditched my sucktastic module as quickly as I could manage.)
• Buying software is cheaper than writing it. (OK you don't pay for software from CPAN, but the idea is the same--perhaps strengthened since the libraries are free.)

If you didn't understand these things already, the response to your first post should have informed you of these issues adequately. Continuing to belabor these points is just a waste of time.

It would be nice to move on and see some critique of your ideas and methods here. Sadly, I don't think this is likely--posts about "My exciting new templating package" touch a raw nerve in these parts.

I don't fully understand your code examples and don't have the tuits to give them the attention they deserve. However, your stated goals appeal to me (a perlish templating module that does templating only and integrates seamlessly with other perl libraries).