Saving compile time by running subroutines as separate files with 'do'

[George_Sherston]

I'm sure this is a bad idea, but I can't see why.

I have a CGI which, depending on user input, does one of a guzillion different things. Each of these things gets done by a subroutine. So for example with subroutines foo, bar and baz, they might be called thus:

if ($Action eq 'foo') {
    &foo;
}
elsif ($Action eq 'bar') {
    &bar;
}
else {
    &baz;
}

But as far as I can see this means that whenever the CGI is called it compiles all the different subroutines, even though it actually only ever runs one of them. And this is true whether the subs are in a lib or in the script itself.

Then I started listening to The Voices. And what they said was, put the code from each subroutine in its own file, call the files foo.pl, bar.pl, baz.pl, and call them with

if ($Action eq 'foo') {
    do 'foo.pl';
}
elsif ($Action eq 'bar') {
    do 'bar.pl';
}
else {
    do 'baz.pl';
}

Now, I can see one reason why this is bad: errors don't show up so easily - you just get a blank screen if foo.pl is n.b.g. But actually that's not too bad, because they're all fairly stable scripts that don't need to change much - and if I'm getting errors, that's a bigger problem than just not being able to track them very easily.

I can't see any other reason why this is bad... but it feels bad. It may be that I've made a mistake about how compiling works and this approach doesn't save any time. I don't think so, but request enlightenment. On the other hand, if it does work - why is it the wrong thing to do?

§ George Sherston

[MZSanford]

just a thought, could you put them in a module and use AutoLoader ?

better on maintainability, but gives the compile-as-needed savings.

---

"They shall not overcome. Whoever told them that the truth shall set them free was obviously and grossly unfamiliar with federal law."
    -- John Ashcroft

[clemburg]

Basically, I doubt that the compile time reduction for the code will give you a significant speedup. And the separation of the code in several files organized by the control structure you suggest will probably expose some interdependencies of the split-out subroutines that you then need to take care of.

OTOH, what you suggest could be the start of real modularization (tm), meaning: why don't you do it the obvious way and collect your subroutines into logical groups (modules). Then you can use the built-in magic of Perl with AutoLoader and AutoSplit.

You might also want to have a look at the Apache Performance Tuning Guide.

Christian Lemburg
Brainbench MVP for Perl
http://www.brainbench.com

[tye]

I'm not sure why everyone is so against your proposal. I'm not a big fan of "do file", but this is one of the best uses of it I've seen recently. A better way of doing the same thing is:

if ($Action eq 'foo') {
    require 'foo.pl';
    &foo();
}
elsif ($Action eq 'bar') {
    require 'bar.pl';
    &bar();
}
else {
    require 'baz.pl';
    &baz();
}

but both versions boil down to the same thing if that code will never get executed more than once in the same run. (If that code gets executed more than once then the do version can compile the external code more than once which will be slower and probably give you warnings.)

And both versions will save the CPU time and memory required to compile two sets of subroutines. If the sets of subroutines are fairly large, then that could be quite a savings (for a short-running script like a CGI is likely to be).

Something like this even has some advantages over any implementations of autoloading that I have seen. For example, using the debugger on this code will be a fairly normal task while using the debugger on autoloaded code usually doesn't work very well (you can make it work quite well, but I haven't seen any implementations that bother to do that). For same reason, autoloaded code that reports source code line numbers says things like "at (eval 14) line 5", which isn't helpful.

With the require approach, you can put something like:

if(  $ENV{DEBUG}  ) {
    require 'foo.pl';
    require 'bar.pl';
    require 'baz.pl';
}

at the top of your script and have errors reported sooner rather than later during testing.

Finally, please don't use &foo;. See (tye)Re: A question of style for more on that.

        - tye (but my friends call me "Tye")

[C-Keen]

It just does not look bad, it is bad under some conditions. As long as you are not using the do statement in a loop construct you should be fine.
But to make you feel better you should use require instead, which provides you with some error checking too!

So long,
C-Keen

[broquaint]

Firstly, don't ever listen to The Voices. Ever. Secondly, according to my fleeting knowledge of do won't all these be done at runtime, which isn't quite so fleet-footed as compile time? And if you're *really* that worried about the slippery sands of clocksecond saving, then you may want to invest in other areas of time saving such as C :o)
HTH

broquaint

[ichimunki]

It feels wrong because it is wrong. :)

The way to make your CGI faster is to have it compile once and stay in memory-- hence mod_perl and the like. No other optimization can beat that.

But your idea for a way of working is interesting, and exists for a reason. Probably just not this reason. It allows for things like Caillte's example of a program that is stored non-traditionally, or even mutates itself without having to restart. It allows existing scripts to be called from new scripts without having to incorporate their internals into your new script. It probably allows for lots of things, I just can't see using it to make a CGI load faster-- since it is fastest to not load the CGI at all (or, really, to load it once and leave it there).

And if your CGI is so huge that it is noticable each time it loads and mod_perl is just not an option, maybe you could break your large script into separate smaller scripts, isolating common elements into a module that can be used from each script. But in practice that may not be any more fun that what you've proposed.

[Caillte]

Like everyone else, I doubt that there is any speed saving. do() is like scalar eval `cat filename` according to perldoc and this leads me to believe that, while compile time may be quicker, runtime will actually be slower due to the overheads needed to set up the sub call. However, there are some obscure advantages to this method; I wrote a program a little while ago that called blocks of perl code from a database upon demand and eval'd them. This was slow and hard to debug but allowed large parts of the program to be edited at runtime.

$japh->{'Caillte'} = $me;

[merlyn]

I don't see anyone else here yet having mentioned SelfLoader. I wonder if anyone will. {grin}

It has most of the benefits of AutoLoader without the complexity of many separate files.

-- Randal L. Schwartz, Perl hacker

[George_Sherston]

Very interesting; thankyou. I'm going to look into Autoloader. I also take broquaint's point - implication being a view frequently expressed by dragonchild that the most efficient to way to increase performance of perl scripts is


$RAM->buy('more');

But I have a supplementary question. I've heard a lot of good things about CGI::Application, but as far as I can make out it doesn't use Autoloader. Does anybody know whether it solves the redundant compilation problem another way? And if not, can one bolt Autoloader onto it?; and also is there a good reason why it doesn't use Autoloader?



§ George Sherston