Tommy has asked for the wisdom of the Perl Monks concerning the following question:

Should we be using Autoloader anymore?


Historically, the use of Autoloader (and autosplit) was to save time loading up libraries of code by tucking away the lesser-used library features into files that got pealed off the final installation of the library and automagically loaded up by perl when a call for such an autoloaded method was made.

The upside was that you saved load/compile time. The downside is that it is expensive to load up these files on demand with all the overhead incurred in such an operation, especially on a one-by-one basis.

What's Your Opinion?

So now that I've told everyone what you already know, I'll get to the point of my question:

It seems like the computational savings historically provided by Autoloader are quickly becoming irrelevant. I have made use of Autoloader in some of my modules and perhaps that has helped folks who used them in their early days, some ten years ago... but nowadays? Hmmmm. I think it probably does not.

So I am asking for your opinions on the subject; to me it seems like a logical decision to dump Autoloader (and autosplit). Do you see any reason(s) to the contrary? For a module that's only 2500 lines, does it make any sense at all to autosplit off 1000 of them anymore? Tests I've run on modern consumer-grade machines show extremely negligable gains, and on enterprise servers, the savings would be practically zero.

Are there reasons why we would continue to use Autoloader? What do you think, monks?

$ perl -MMIME::Base64 -e 'print decode_base64 "YWNlQHRvbW15YnV0bGVyLm1lCg=="'

Replies are listed 'Best First'.
Re: Use of AutoLoader in Modern Times (future)
by tye (Sage) on Nov 16, 2012 at 00:27 UTC

    Thankfully, AutoLoader and similar modules have been becoming less widely-used. Having repeatedly seen code that takes way too long to load all of the tons of modules that eventually get used, my experience is that the slow loading is always due to a bunch of initialization that is being done too soon, not due to the time required to read and compile the code for subroutines. Though I've often seen people making the assumption that it was the time to read and compile the modules' code that was to blame. (See Devel::Init for my suggestion for how to avoid doing even slightly costly initialization too soon, or too late.)

    So I suspect that the creation of modules for delaying the loading of module code may have been significantly motivated by flawed assumptions.

    The worse problem with AutoLoader is that it is applied to a module via inheritance, which was just a horrible idea. Autoloading is not a property that should be inherited. If class X autoloads and class Y inherits from class X, it is simply a mistake for class Y to get impacted by class X's decision to use autoloading.

    This is made even worse by AutoLoader giving a particularly bad error message for the case of a mispelt method name. Now calling Y->foo() when I meant to call Y->food() tells me that I installed the Y module incorrectly because I clearly left out the Y/ file (when all that I left out was the letter 'd' in my code).

    And this is made even worse by the twin design mistakes of having DynaLoader 1) use AutoLoader, and 2) be used via inheritance.

    And aspects of this are also made worse by the fact that Perl has great support for using a module that makes changes that have to be made at compile-time (because they impact the syntax of the following code) but has awkward support for using modules that don't need to make changes at compile time.

    It would be very nice for require to be enhanced so that it can be used more like use and also be more useful for loading OO modules. That is, require should be extended so that it can be used both of these ways:

    # You must 'use' here, since new syntax is being defined: use My::Exceptions qw< try catch >; # New syntax: try { ... } catch { + ... } # You can 'require' here, since no impact on syntax: require My::Utils qw< min max zip >; # Parens required (no new syntax) when calling min(), max(), or zip(). my $logger = require My::Logger( -trace => 1 ); # Object returned (maybe a factory)!

    And then it should be a decision based on the environment as to whether 'require' actually loads the module code at run-time or at compile-time. I'd default to loading at run-time and then having a "test dependencies" step that happens relatively frequently (at least before deploying and after deploying) that specifies "load all dependencies at compile-time" and then pretty much just exits.

    I plan to make a proof of concept of "better 'require'" under a different name ('load'?) but also to push for Perl's own 'require' to get better in parallel (in my "copious free time").

    - tye        

      Module::Runtime's use_module function may be to your liking...

      use_module('My::Utils')->import(qw< min max zip >); my $logger = use_module('My::Logger')->new(-trace => 1);

      Update:... some people may well consider this insane but...

      BEGIN { package PerlX::qm; no thanks; sub import { my ($me, $name) = @_; require Module::Runtime; require PerlX::QuoteOperator; PerlX::QuoteOperator->new->import($name || "qm", { -parser => 1, -with => sub ($) { Module::Runtime::use_module(grep /\w/, split /\s/, $_[ +0]) }, }, scalar caller); } }; use 5.010; use strict; use warnings; use PerlX::qm; say qm( Math::BigInt 1.000 )->new('4660')->as_hex; say qm( Math::BigFloat )->new('74565')->as_hex;

      PerlX::qm defines a quote-like operator (like qq, qw or qx) for module names. Quoted modules are loaded at runtime, and can be followed by an optional version number.

      perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'
Re: Use of Autoloader in Modern Times
by tobyink (Abbot) on Nov 15, 2012 at 20:52 UTC

    There's no point in them these days. Better to split your code into a few logical modules and then require them on-demand.

    Not to be confused of course with AUTOLOAD which was originally designed to support Autoloader-type applications, but has become a useful general purpose facility.

    perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'
Re: Use of Autoloader in Modern Times
by bulk88 (Priest) on Nov 15, 2012 at 22:28 UTC
    A require is a require. Whether you do it at "startup" or "runtime" is irrelevant over the lifetime of the process. This is not C or Java, your whole program is not recompiled from a single require. There is memory savings to be had from not keeping so many subs and globs in memory.
      Whether you do it at "startup" or "runtime" is irrelevant over the lifetime of the process.

      ... until you run a preforking server and want to take as much advantage of COW as you can.

        Which is one of the reasons why "load dependency at compile-time" vs "load dependency at run-time" should be decided by the environment (when possible). Modules that define new syntax must be loaded at compile-time. A huge fraction of modules don't need to be loaded at compile-time so it can be very handy to allow the invoker to pick which behavior.

        Almost all unit tests should delay loading modules since the process isn't going to live very long and isn't going to use the vast majority of functionality eventually reached by the dependency tree (so no reason to wait for all of the too-early initialization over and over). But you should have at least one unit test that specifies "load all dependencies at compile-time" so that you can very quickly tell if any dependencies are missing, even if you haven't figured out how to exercise every single one of them.

        Pre-forking servers should load dependencies at compile-time in the parent but at run-time in the children.

        Lots of command-line tools (especially in a corporate software development house) can benefit from loading dependencies at run-time, so that I can do something useful with the frobnitz repository management subsystem from my laptop without having to install all of the PDF, XML, e-mail, GD, and plotting modules despite parts of the frobnitz repository needing to know how to send e-mails with PDF and/or XML attachements, some of which contain bar charts as png files (while also not having to constantly groom the dependency tree to make selected dependencies optional or at run-time to get that benefit).

        - tye        

        ...until you run a preforking server and want to take as much advantage of COW as you can.

        Could you expand on that a bit? How does copy on write and preforking incline one to use autoloading?

        My own personal preference is to avoid it like a plaguish thing, but even my most massive module isn't on the same order as some CPAN monsters.

        (Update: fix the italics tags)

      ...Right, but it's an extra I/O op (or more) just to pull in the sub out of the .al file, each on an individual basis...

      Are the savings worth the cost?

      $ perl -MMIME::Base64 -e 'print decode_base64 "YWNlQHRvbW15YnV0bGVyLm1lCg=="'

        Presumably that will depend on the individual script's use case.

        If only 1 of every 1000 runs will perform the conditional require, then it's probably negligible. If 999 of every 1000 runs needs it, then it is probably not negligible.

        If in doubt, benchmark!