Benchmarks for module compilation time

[itub]

We all know that using a module adds a little bit to the startup time of a Perl program, because the module has to be loaded and compiled on the fly (added: or linked, in the case of XS extensions). I was wondering exactly how much the cost for using a module is, so I ran some simple tests with a bunch of popular modules.

Basically, I timed how long does it take to run this program 11 times and took the average, discarding the first point because it usually takes longer than the other 10 due to caching:

perl -M$module -e1

The average times, in seconds (on a 2.8 GHz Pentium 4 with Linux), were:

diagnostics             0.100
Archive::Zip            0.092
Class::DBI              0.070
Archive::Tar            0.060
Template                0.036
YAML                    0.034
DBI                     0.034
Lingua::EN::Inflect     0.030
Math::Trig              0.023
CGI                     0.023
XML::SAX                0.020
XML::Simple             0.020
POSIX                   0.017
Data::Dumper            0.016
warnings                0.007
strict                  0.004

all of the above        0.390
none                    0.003

Note that the times are not additive: the sum for all the individual modules is 0.586 s, which is much larger than using all of them at the same time (0.390 s). I think this is partly due to the overhead of starting perl each time (which could account for about 0.045 s), but mostly because of common module dependencies.

To summarize:

• Running a "null program" takes 3 ms on my system.
• Using strict and warnings doesn't cost that much in terms of startup time.
• Loading a module typically takes about 20-30 ms. (This depends on which modules you consider typical, of course, and this time includes the dependencies as well.)
• The "heaviest" modules I tested take up to 100 ms.
• Curiously, the heaviest module was diagnostics, while some other modules that are popularly regarded as heavy were not that bad.

[szabgab]

that's nice

When you are looking at the bigger picture you might want to take in account (and if that is interesting measure it too) that some of the modules rely heavily on AUTOLOAD and create some of their own functions or load additional modules in run time and based on the flow of your program.

Specifically CGI.pm is mostly a huge string called $AUTOLOADED_ROUTINES from where the rest of the code extracts the functions as they are needed.

[itub]

POSIX is another example of a big module that uses AUTOLOAD. I wanted to test that one specifically because sometimes I hear that people are afraid of using it because it is "too big". However, for the "null use" of POSIX, it's one of the fastest to start up. I often use it just because I need a little function such as floor or ceil.

[szabgab]

speed is one thing.

what might be also interesting is the memory footprint without POSIX, in "null use" and when using 1-2-3 functions.

[simonm]

Very nice, thanks!

How would you feel about posting your benchmarking code so we can try this on other platforms?

[itub]

There was not really code as such, just a quick and dirty series of shell commands. It went something like this:

#!/bin/bash
 
modules="Archive::Tar Archive::Zip CGI Class::DBI DBI Data::Dumper Lin
+gua::EN::Inflect Math::Trig POSIX Template XML::SAX XML::Simple YAML 
+diagnostics strict warnings"

export TIMEFORMAT="%R"
rm -f table
for m in $modules; do
    time perl -M$m -e1;
    for i in `seq 1 10`; do
        time perl -M$m -e1;
    done &> $m
    echo -ne "$m\t" >> table
    awk '{t+=$1} END{print t/10}' $m >> table
done
 
sort -k2 -rn table > table.sorted

The part that computes "all" and "none" is left as an exercise to the reader. ;-)

I hope I won't be downvoted for using shell instead of Perl! ;-) Sadly, this makes the test less portable.

[radiantmatrix]

Hm, it wouldn't be very hard to do this with Perl. I can think of two approaches: either invoke perl via system(), or use string-based eval.

The following can be used like:
# this_script.pl [--eval] [--perl /my/perl/interpreter] module module2 module3

use strict;
use warnings;
use Time::HiRes;
use Benchmark ':hireswallclock',':all';
use Getopt::Long;

my $PERL = 'C:\\Perl\bin\perl.exe';
my $use_eval = 0;

GetOptions ( 'perl=s' => \$PERL, 'eval' => \$use_eval );

my %testhash = map { 
   $_ => ( $use_eval ? "eval 'use $_'" : "system('$PERL', '-M$_ ', '-e
+1')" )
} @ARGV;

timethese( ($use_eval ? 10000 : 1000), \%testhash );

This one was created to work with ActiveState Perl on Windows. Pass the --perl parameter to use it on other platforms.

Update:itub confirms my suspicions that eval isn't as accurate a test as the system route. Thanks for the explanation of why, itub! It's worth noting that the system call is the default approach in the code above.

Updates:

• 2005-07.Jul-26 : clarified preferred use of system over eval, thanks to itub

[Anonymous Monk]

Using strict and warnings doesn't cost that much in terms of startup time.

The load time of strict.pm or warnings.pm isn't very interesting. To get an idea of the impact of using strict.pm, one should measure the load time of a (large) program and/or module with, and without using strict. I still expect to difference to be tiny. The impact of using warnings.pm is harder to measure, due to its runtime effects.

[itub]

I know, that's why I qualified the statement with "in terms of startup time". ;-)

Added: ikegami and the Anonymous Monk are right, of course; I misunderstood the parent node.

[ikegami]

That's still ambiguous. You meansured the effect on the module's startup time, but not on perl's startup time for strict and warnings since the effect varies depending on the script begin loaded.