Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked

Creating huge files with the Template Toolkit

by holli (Abbot)
on Jul 15, 2005 at 11:52 UTC ( #475184=perlquestion: print w/replies, xml ) Need Help??

holli has asked for the wisdom of the Perl Monks concerning the following question:

In the near future I will have to create huge XML files (along the lines of 4-6 GB (yes, Gigabytes)) from various data sources. I was thinking to use the Template-Toolkit for this task. I played around with it and like the way it works.

However, after some deeper studying of the manual I realized, that all of the templates output gets concatenated to a single string. That of course will cause a huge memory footprint when dealing with output files that big. So I am now thinking about two ways:
  1. Find a way to "flush" (emptying the output string after printing it) the template in time so, I can spare memory.
  2. Use another Templating engine that does not have this limitation.
I would prefer the first option, since I am now familiar with the TT and really like it's syntax and plugin mechanism.

I'd be happy if you share your insights with me.

holli, /regexed monk/

Replies are listed 'Best First'.
Re: Creating huge files with the Template Toolkit
by (anonymized user) (Curate) on Jul 15, 2005 at 12:35 UTC
    The object-oriented nature of TT suggests a possible strategy:

    - Identify the method that is handling the strings in the way you want to avoid.

    - Create your own class that inherits from the TT class.

    - In your own class, override the offending method to give you control over I/O buffering - perhaps also using additional classes such as IO::File and IO::String

    Hope this helps.


    One world, one people

Re: Creating huge files with the Template Toolkit
by bravenmd (Sexton) on Jul 15, 2005 at 14:09 UTC
    I have used the XML::Writer utility from CPAN to create similar sized XML documents. I have never run into any memory issues with it but that does't mean they don't exist. Take a look and see if it may help you. The module is very easy to use and I found great value in it.
      Indeed. Do you *really* need to use TT? I really can't think about a reason to use it, since XML::Writer would deal better with special characters, encoding and so on...
Re: Creating huge files with the Template Toolkit
by thunders (Priest) on Jul 15, 2005 at 14:29 UTC
    I found this thread on the mailing list. Click here

    It sounds like a bit of a hack, basically you alter the $OUTPUT token for the perl code TT generates.

    In Template::Directive it is defined as:
    $OUTPUT    = '$output .= ';

    You can set it to something like this
    $Template::Directive::OUTPUT = 'print ';

    On the thread Andy notes that this will break WRAPPER and lists a few other caveats.

      can't you just output to a filehandle? (note that I'm not a TT expert, I'm just wondering)
Re: Creating huge files with the Template Toolkit
by perrin (Chancellor) on Jul 15, 2005 at 12:54 UTC
    Someone did this years ago and posted it to the TT list. Check the archives.
      Thanks for the info. I must admit I am absolutely unfamiliar with mailing lists. I am currently googling, but I would appreciate if you could assist me a bit further :)

      I have found this thread. I will email myk and see if I can get a hand on his patch.

      holli, /regexed monk/
Re: Creating huge files with the Template Toolkit
by mikeraz (Friar) on Jul 15, 2005 at 16:26 UTC

    Curiosity time. Just what are you doing to create 4GB xml files?

    Be Appropriate && Follow Your Curiosity

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://475184]
Approved by marto
Front-paged by tlm
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (5)
As of 2020-10-19 21:54 GMT
Find Nodes?
    Voting Booth?
    My favourite web site is:

    Results (207 votes). Check out past polls.