Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Efficient Perl Programming

by Fastolfe (Vicar)
on Nov 08, 2000 at 22:42 UTC ( #40593=perlquestion: print w/replies, xml ) Need Help??

Fastolfe has asked for the wisdom of the Perl Monks concerning the following question:

Are there any documents out there on getting the most efficiency/optimization out of your Perl? A lot of regexp optimizations, the keys(%hash) = $large_value trick, and a million other things to get the most speed out of your Perl are usually/occasionally documented (especially here in various nodes), but don't seem to be readily available in a central document.

Is there anything out there that fits this criteria?

Replies are listed 'Best First'.
(redmist) Efficient Perl Programming
by redmist (Deacon) on Nov 08, 2000 at 23:35 UTC
RE: Efficient Perl Programming
by KM (Priest) on Nov 08, 2000 at 22:54 UTC
Re: Efficient Perl Programming
by little (Curate) on Nov 08, 2000 at 22:57 UTC
    I do personally like "Effective programming perl" (J.N.Hall, R.Schwartz, Addison-Wesley, look up for samples or Book Reviews for reviews by arturo and jbardhan) very much. On the other hand it's at least to me always interesting to read merlyn's columns (always tricky and full of magic) and to read nodes from the past, especially in the code section or trying to read and to get module sources, but that's still hard to me. :-)

    Have a nice day
    All decision is left to your taste
RE: Efficient Perl Programming
by arturo (Vicar) on Nov 08, 2000 at 22:48 UTC

    Chapter 8 of the second ed. Camel has a bunch of things (and on different kinds of efficiency). Other n' that I don't know of a central repository. Perhaps an addition to the Tutorial page is in order ...

    Philosophy can be made out of anything. Or less -- Jerry A. Fodor

Re: Efficient Perl Programming
by tedv (Pilgrim) on Nov 08, 2000 at 22:51 UTC
    Yes, there are, but I mostly know of ones in print. The O'Reilly Perl 5 book has a two good sections on efficiency and coding style on pages 537-550. The O'Reilly Regular expression book has a lot of information scattered in it about what makes particular regular expressions fast or slow. (In particular, do not use $`, $&, or $'.)

    Naturally, the entire Perl Cookbook is a resource on how to do particular tasks efficiently, and makes a good starting point. Lastly, there is this great site about perl where you can search for all things perl related. If you had a particular question, I'm sure you could find the answer there. :)

(Aighearach) Re: Efficient Perl Programming
by Aighearach on Nov 10, 2000 at 18:14 UTC

    What I would advise if you want to get your program running faster, is to use Devel::OpProf. This ties right into the perlfaq 3 question, "How do I make my Perl program run faster?" The advice given is to improve your algorithm. Okay, but how to do that? One of the best ways to compare different ways of doing things is to look at the resources they take up. will tell you how fast a code fragment runs, but that information isn't very portable. On a machine with lots of RAM, you might be developing programs that are very fast, but will generally lag on other people's machines. Devel::OpProf lets you identify these bottlenecks, even when they aren't slowing you down.

    As an example, lets consider that we want to make a list of 1000 elements, with each element set to 1.

    #!/usr/bin/perl -w use strict; use Devel::OpProf qw( profile print_stats zero_stats ); use Benchmark qw( timethese ); profile(1); print "*** one() ***\n"; my @one = one(); print_stats(); zero_stats(); print "\n*** two() ***\n"; my @two = two(); print_stats(); zero_stats(); print "\n*** three() ***\n"; my @three = three(); print_stats(); zero_stats(); print "\n*** four() ***\n"; my @four = four(); print_stats(); zero_stats(); timethese( 0, { test_one => 'one()', test_two => 'two()', test_three => 'three()', test_four => 'four()' } ); sub one { my @list; for ( my $i = 0; $i < 1000; $i++ ) { $list[$i] = 1; } return @list; } sub two { my @list; for ( 0..999 ) { $list[$_] = 1; } return @list; } sub three { my @list = map { 1 } (1..1000); return @list; } sub four { my @list; @list[0..999] = (1) x 1000; return @list; }
    On my machine, this outputs:
    *** one() ***
    private variable         3002
    constant item            2003
    next statement           1009
    private array            1004
    numeric lt (<)           1001
    logical and (&&)         1001
    scalar assignment        1001
    preincrement (++)        1000
    iteration finalizer      1000
    array element            1000
    pushmark                 8
    glob value               4
    subroutine entry         3
    conditional expression   1
    list assignment          1
    block entry              1
    array dereference        1
    loop entry               1
    loop exit                1
    return                   1
    print                    1
    *** two() ***
    next statement           2009
    private array            1004
    constant item            1003
    foreach loop iterator    1001
    logical and (&&)         1001
    array element            1000
    scalar variable          1000
    iteration finalizer      1000
    scalar assignment        1000
    pushmark                 9
    glob value               5
    subroutine entry         3
    block entry              1
    list assignment          1
    conditional expression   1
    array dereference        1
    foreach loop entry       1
    loop exit                1
    return                   1
    print                    1
    *** three() ***
    null operation           1002
    constant item            1002
    map iterator             1000
    block                    1000
    pushmark                 11
    next statement           8
    private array            4
    glob value               4
    subroutine entry         3
    array dereference        2
    list assignment          2
    map                      1
    block entry              1
    conditional expression   1
    return                   1
    print                    1
    *** four() ***
    pushmark                 12
    next statement           9
    private array            5
    glob value               4
    constant item            4
    subroutine entry         3
    array dereference        2
    list assignment          2
    repeat (x)               1
    conditional expression   1
    null operation           1
    array slice              1
    block entry              1
    return                   1
    print                    1
    Benchmark: running test_four, test_one, test_three, test_two, each for at least 3 CPU seconds...
      test_one:  3 wallclock secs ( 3.16 usr +  0.00 sys =  3.16 CPU) @ 274.68/s (n=868)
      test_two:  3 wallclock secs ( 3.25 usr +  0.00 sys =  3.25 CPU) @ 359.08/s (n=1167)
    test_three:  3 wallclock secs ( 3.01 usr +  0.00 sys =  3.01 CPU) @ 408.64/s (n=1230)
     test_four:  3 wallclock secs ( 3.27 usr +  0.01 sys =  3.28 CPU) @ 501.83/s (n=1646)

    So, while Benchmark could tell us which is faster, it doesn't really tell us why. Devel::OpProf shows us that it is because of the number of temporary variables, &&'s, assignments, etc.

    (while not really relevant to the point of this post, I should point out that probably a better idea than the slice in #4 is to just my @list = (1) x 1000;, because it is more readable. But, you are welcome to profile them to see if there is an algorithmical difference... ;)

    Paris Sinclair    |    4a75737420416e6f74686572    |    205065726c204861636b6572
      I was a little more interested in finding documentation (perldoc-style if nothing else), or a niche waiting to be filled by some useful documentation, on general optimization tips (e.g. use !/\S/ instead of /^\s*$/ and why). This information might still be useful to others though. Thanks.
        Here's one rule you might like: It is almost always a mistake to use the non-greedy regex quantifiers like *? and +?.

        Typical code looks like this:

        $churchill = qq{"If I were your husband," he said, "I should drink + it."}; while ($churchill =~ /"(.*?)"/g) { print $1, "\n"; }

        The purpose of the (.*?) is to capture a quoted part of the string in $churchill. Beginners usually try (.*) which doesn't work, because it captures everything from the first quotation mark to the last, including the non-quoted part in between. So then they ask how to fix this and are advised to use (.*?) instead. This does work, but it's much slower than it needs to be. A faster solution is:

        while ($churchill =~ /"([^"]*)"/g) { print $1, "\n"; }

        This says that what you're interested in is a quote character, followed by a sequence of characters that are not quotes ([^"]) followed by another quote. The description of what you want is more exact, and it enables Perl to do the matching more efficiently.

        So a good rule of thumb is to avoid .*? wherever possible and to use something like [^"]* instead when you can.

          A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Efficient Perl Programming
by japhy (Canon) on Nov 09, 2000 at 00:45 UTC
    I've been wanting to write a book of some sort, a compendium of elegant Perl idioms. Were that this book existed, I would point you to it. ;)

        I don't think that "a compendium of elegant Perl idioms" will be a good description of the Perl Advanced Techniques Handbook. When people talk about 'idioms' they usually mean one- or two-line snippets like the one fastolfe mentioned that pre-extends a hash. But PATH isn't going to be about that at all.

        The idea of PATH is that there are a lot of powerful programming techniques that are possible in Perl but not in other languages that Perl programmers are familiar with. Since Perl programmers haven't seen these techniques before, they don't know how to use them, what they are good for, or even that they exist, and they are letting a lot of the power and expressiveness of Perl go to waste. The techniques are not little things like pre-extending a hash. They are much bigger ideas that apply to the organization of entire programs, ideas on the scale of 'object-oriented programming'.

        One example is the idea that in Perl you can write a function that manufactures other functions. Instead of writing a lot of similar functions in the source code, you instead write one function which, when called, generates the function that you actually want to use and returns it. By invoking this 'function factory' with different arguments, your program can manufacture as many functions as it needs to without your having to guess in advance what all the functions will need to do.

        Perl's own sort operator is a limited example of this. If sort only sorted lists alphabetically, it would only bone ten-thousandth as useful as it is. But instead, it gets an argument, supplied by the programmer, which tells it how to compare two list elements. In effect, this extra argument transforms sort into a different function, which sorts lists in the way that the programmer specified; if you give it a different argument, you get a different kind of sorting function back out.

        This is tremendously useful in the case of sort, but most Perl programmers don't realize that they can apply the same techniques to their own functions in many similar circumstances and make their own functions much more useful and general than they would have been otherwise.

        Anyway, if you're interested, the correct URL is

        Mine wouldn't be a link to ;)

        Maybe it doesn't. I don't know -- I've never heard of this handbook of which you speak. I have heard a couple of his talks on the matter. So if he's making a compedium of them, that would be super.

RE: Efficient Perl Programming
by Dominus (Parson) on Nov 11, 2000 at 19:15 UTC
    You may want to take a look at Optimization of Computer Programs in C. It's ostensibly about C, but actually most of the optimization techniques it discusses (such as loop expression hoisting) are applicable in any language, including Perl.
Re: Efficient Perl Programming
by metaperl (Curate) on Nov 09, 2000 at 20:34 UTC is the URL to MJD's book-in-progess: "Perl Advanced Techniques Handbook"

    Also, are you going to wait until Perl 6 to write this book?

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://40593]
Approved by root
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (4)
As of 2021-09-18 18:15 GMT
Find Nodes?
    Voting Booth?

    No recent polls found