Optimizing report generation

[Anonymous Monk]

I have a web application that generates large, detailed PDF reports. The reports are created through a combination of Perl and Java. Perl is used to generate SVGs for each page of the report. The Perl script then calls two different Java programs via system and backticks. The first java program converts each SVG to a PDF. The second merges all of the PDFs into a single document which is returned to the Perl script. Finally, the Perl script stores the result in a database using DBI.

It's a complicated system, but it works well for generating the reports that we need. The problem is that it is slow. I think there is a significant amount of time being spent on startup, both for Perl as well as for each of the system commands. I can pretty easily make the Perl persistent. Is there a way to save time on the system commands that execute the Java programs? Unfortunately, replacing the Java isn't a viable option at this point.

[MidLifeXis]

I think there is a significant amount of time being spent...

Don't "think there is", profile, and know there is. Something as coarse grained as logging the start / stop times of each of the steps (see Time::HiRes) may be enough.

[thomas895]

Also, large amounts of binary data(the PDFs) are best stored in files and not databases. While it might be handy, it can be rather slow, so you might want to try file-based storage.

[bulk88]

Your question is better suited for a java forum since as you said, you can't replace it.

I can't see why you need java. PDFs are a tree-ish with some circular references, text format under the hood. From searching CPAN, I found CAM::PDF, which looks like it can manipulate the COS tree of a PDF. I've never used it personally. In other languages, I've written code that uses a PDF as a template, and using the PDF's COS tree format, fills in the data into the new PDF, not unlike HTML. Instead of java, you can try standalone text mode pdf tools, they will be faster than java since they are written in C/C++.