HTML::Template uses the idea of separation of code and presentation. I happen to believe in this idea very strongly. Unfortunately, the present method HTML::Template uses to fill in everything is to store all information in variables that are then later used to display the content. This is normally ok, however, for code where there is a huge amount of data, this can become a problem. HTML::Template actually has a method that allows the template to print directly to an output stream as it's being generated, namely by doing $template->output(print_to=>*STDOUT). When I saw this feature, I immediately became very excited as this would allow some of the large-data producing templates I use to execute without eating up a huge chunk of memory while being generated. I was a bit discouraged when I discovered that TMPL_LOOP didn't enjoy the same optimization. No matter what, it would store straight to a variable and then it would in one big chunk be printed to the filehandle.

Then I thought to myself, "Jim," (...which isn't my's actually Matt, but it's ok for this story...), "imagine a world where HTML::Template's TMPL_LOOP parsed the data to be displayed as it was being generated, skipping a variable and going straight out. Will this make the overhead of using HTML::Template go away for the most part?" The chase was afoot.

First, I needed something that would allow me to iterate over my data while parsing it. I found Tie::Array::Iterable, but it's only for lists and arrays. This led me to attempt to create a tie module that would allow for me to DWIW. I wanted something that took advantage of the way foreach works.

A note on how it does work: foreach will check the length of the array it is trying to pull from before it attempts to grab the next item. If the length of the array is ever equal to the next index it is going to pull, it stops. In other words, just like your usual for loop but with all the magical index-tracking stuff implied.

I checked inside of HTML::Template::LOOP::output() (which is what is called to get the TMPL_LOOPs to work) and found a foreach. My heart was racing with excitement. I constructed my tie module, created a test script, and tried it out. But alas, there was a problem. If I didn't specify a length with my tied parser and it used the default "check until determine_end", it'd just grab the first one. It turned out this was due to the HTML::Template parser which performs [@{$passedref}] on the reference passed, creating a copy of our tied array. The problem here, though, is that it checks the length only once, never allowing the lengthening process to occur. So what can we do? Well, here's what I did.

I pulled out my favorite editor (who is Terry O'Hara, but not that kind of editor) to make a few changes to HTML::Template. First up was to allow a tied array to be used when its length varies. Also, I had HTML::Template::LOOP::output() take a reference to $result from the caller as a parameter yet did it in such a way that it would still operate as before (you case...ummm...I was brainwashed in boy scouts. I must always be prepared). I used HTML::Template 2.6. These are the alterations I made:

2467c2467 < $param_map->{$param}[HTML::Template::LOOP::PARAM_SET] = [@{$va +lue}]; --- > $param_map->{$param}[HTML::Template::LOOP::PARAM_SET] = $value +; 2612c2612 < eval { $result .= $line->output($x, $options->{loop_context_ +vars}); }; --- > eval { $line->output($x, $options->{loop_context_vars},\$res +ult); }; 2885a2886,2888 > my $result; > my $blank = ""; > $result = shift or $result = \$blank; 2906c2908 < $result .= $template->output; --- > ${$result} .= $template->output; 2914c2916 < return $result; --- > return ${$result};

Now for my tie module.

package Tie::Array::SubIterator; use base Tie::Array; sub TIEARRAY { my $class = shift; my %in = @_; $in{'length'} = -1 unless $in{'length'} && $in{'length'} =~ /^\d$/; die "Expected coderef for next option" unless ref($in{'next'}) eq "C +ODE"; die "Expected coderef for determine_end option" unless !defined($in{ +'determine_end'}) or $in{'length'} < 0 or ref($in{'determine_end'}) e +q "CODE"; $in{'determine_end'} = sub { return defined(+shift->{'last_result'}) + ? 0:1; } if (!$in{'determine_end'} && $in{'length'} < 0); $in{'last_counted'} = 0; $in{'ended'} = 0; my $self = bless \%in, $class; eval { $self->{'last_result'} = $self->{'next'}->($self,0) if $self- +>{'length'} < 0; }; $self->{'ended'} = 1 if $@; return $self; } sub FETCH { my $self = shift; return $self->{'next'}->($shift,@_) if ($self->{length} > -1); my $last = $self->{'last_result'}; eval { $self->{'last_result'} = $self->{'next'}->($self,++$_[0]) unl +ess $self->{'ended'}; }; $self->{'ended'} = 1 if $@; return $last; } sub FETCHSIZE { $self = shift; return $self->{'length'} if $self->{'length'} > -1; ++$self->{'last_counted'} unless ($self->{'ended'}) || ($self->{'end +ed'} = $self->{determine_end}->($self)); $self->{'last_counted'}; } sub DESTROY { my $self = $_[0]; $self->{destroy}->(@_) if ref($self->{destroy}) eq "CODE"; } 1;

Some may find this module to be fairly disgusting (I know I do), but it allows for a number of options to be passed to it. First, it must have a next option set to a coderef. That's pretty much all you need for it to work, as long as it returns undef when it's finished. Next is passed a copy of the object (which is a blessed hashref) and the parameters passed to fetch. You can also use your own method for determining the end by setting determine_end to a coderef. It should return something that can be evaluated as true when it thinks we should end. Remember that values are prefetched. Check the last_result value of the hashref passed as the first parameter if you would like to see what the next iteration should return. If you wish for only a certain number of iterations to be performed, just set length to some number greater than -1 and your next will be executed that many times. In all truth, the module is still very rough. (Sorry if that was the worst description ever)

And now, a test script.

#!/usr/bin/perl -w use strict; use Tie::Array::SubIterator; use HTML::Template; use Devel::Size qw(total_size); my @array= (1..10_000); sub next { my $self = shift; return undef if $_[0] > $#array; return { num => $array[$_[0]], added => $array[$_[0]]+10, subtracted + => $array[$_[0]]-10 }; } my @old_method; push @old_method, { "num" => $_, "added" => ($_ + 10), subtracted => $ +_-10 } for (@array); print "old: ", total_size(\@old_method),$/; print "new: ", total_size(\@array),$/; tie my @data, "DMN::TieArrayForeach", next=>\&next; my $tmpl = q|<tmpl_var bob> <tmpl_loop data> <br><tmpl_var num> <tmpl_var added> <tmpl_var subtracted><br> </tmpl_loop> <tmpl_var bob>|; my $template = HTML::Template->new(scalarref => \$tmpl); $template->param(data => \@old_method,bob=>"outer area"); my $one = $template->output(); $template->param(data => \@data,bob=>"outer area"); my $two = $template->output(); print "identical$/" if $one eq $two;


old: 2145624 new: 240056 identical

Now I understand that these numbers don't show a true picture of everything that's going on. The point of showing the Devel::Size::total_size of both variables is to show that the amount of memory required is less.

Some notes: a good reason not to use $template->output(print_to=>*HANDLE) is that instead you can call eval { $somevar = $template->output(); } and check $@ instead of catching the errors after you've already printed out a good portion of the page. In other words, this modification (read: hack) isn't for everyone. Also, I'm terribly sorry for the parts where this became more of a blind tour through the deeper recesses of a (read: my) warped mind. However, I hope someone finds this useful.

Finally, I would like to thank Sam Tregar and the other kind folks in the HTML::Template community for giving us a very cool module. Also, thanks to jeffa for guidance and suggestions.

The first rule of Perl club is - use Perl
ith rule of Perl club is - follow rule i - 1 for i > 1

Replies are listed 'Best First'.
Re: On HTML::Template
by Mr. Muskrat (Canon) on Aug 06, 2003 at 13:18 UTC

    A note on how it does work: foreach will check the length of the array it is trying to pull from before it attempts to grab the next item. If the length of the array is ever equal to the next index it is going to pull, it stops. In other words, just like your usual for loop but with all the magical index-tracking stuff implied.

    for behaves the same as foreach. perlsyn has this to say: "The foreach keyword is actually a synonym for the for keyword, so you can use foreach for readability or for for brevity."