In the near future I will have to create huge XML files (along the lines of 4-6 GB (yes, Gigabytes)) from various data sources. I was thinking to use the Template-Toolkit for this task. I played around with it and like the way it works.
However, after some deeper studying of the manual I realized, that all of the templates output gets concatenated to a single string. That of course will cause a huge memory footprint when dealing with output files that big. So I am now thinking about two ways:
- Find a way to "flush" (emptying the output string after printing it) the template in time so, I can spare memory.
- Use another Templating engine that does not have this limitation.
I would prefer the first option, since I am now familiar with the TT and really like it's syntax and plugin mechanism.
I'd be happy if you share your insights with me.