Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Benchmark Arena Proposal

by MeowChow (Vicar)
on Jan 30, 2001 at 02:25 UTC ( #55103=monkdiscuss: print w/ replies, xml ) Need Help??

I've noticed that despite Perl's mostly undeserved reputation as just another "inefficient" interpreted scripting language, most good Perl developers (and especially those who frequent Perlmonks ;), are quite concerned with their code's performance, and often benchmark isolated snippets of code in order to find out how efficiently Perl performs a given algorithm/function, or to catch a glimpse of whether or not Perl internally optimizes some particular syntax. Credit for this is due in no small part to the inclusion of the Benchmark module in the standard Perl distribution, which makes comparing and benchmarking Perl code a relatively trivial, non-tedious, and even fun task.

Perl users, especially beginners, are often surprised at how well Perl optimizes things, and that they don't have to code in round-about ways (as is the case in many other interpreted/scripting languages) to speed up their programs. And, from time to time, people are surprised that an operation which is seemingly efficient is actually dog-slow under Perl (eg. returning multiple elements per map call).

I have, of late, been running many general-interest benchmarks on this and that, and have actually come to the point where I'm naming my benchmark scripts, organizing them by keywords, and providing little blurbs as to what I learned from each benchmark. For example:

# keywords: method sub call overhead # learned: OO-style method invocation is only about 13% slower than d +irect invocation (v5.6) package Foo; use strict; use warnings; use Benchmark qw(cmpthese); sub new { bless {}, shift; } sub meth { my ($self, $val) = @_; $val; } my $obj = new Foo; my @params = (42, 1, 2, { a=> 'silly hashref'}, 'a string'); cmpthese (-3, { method => sub { $obj->meth(@params); }, direct => sub { meth($obj, @params); } });
The point I'm trying to make is not regarding this particular benchmark (although it is edifying to know that method invocation doesn't have very much overhead), but regarding the usefulness of such information for the Perl community as a whole. I have noticed that a substantial portion of SoPW nodes relate to the issue of performance and have some sort of benchmarking in them, often of general interest.

So, I've been contemplating the idea of a "Benchmark Arena", where people could submit categorized benchmarks, and discuss specific performance-related issues. A Benchmark Arena could serve as a communinal repository of information regarding optimal ways of doing such-and-such under Perl, like a Q&A section, but devoted specifically to the issue of code performance. Because Perl offers so-many-WTDI, it is very enlightening to see several WTDI in action, and see how they compare against each other. The benchmark arena would ideally be keyword-searchable, so for example, the benchmark above could be found under the "method", "sub", "call", and "overhead" keywords, while benchmarks regarding map performance would be found by searching for "map", etc. Keyword categorization would allow much greater specificity of search results than would be possible with a plain-text search.

I believe something like this would serve as a very valuable resource to the Perl community at large, and would be ideal for Perlmonks. What have my fellow monks to say about this idea?

Comment on Benchmark Arena Proposal
Select or Download Code
Re: Benchmark Arena Proposal
by KM (Priest) on Jan 30, 2001 at 02:34 UTC
    Ohhh.. I like this idea. It would be good to see/show how to take Perl code and make it more efficient or speedier. But, I would fear it being filled with "Here is my homewo..I mean code. How can I make it faster" type of things. I would like to see it as a "I did this, it sucked, then I did this and it rocks!" Good idea.

    Cheers,
    KM

      Heh ... but how often is the seeker of Exactly the Amount of Perl Wisdom Needed to Answer the Homework Question going to want optimizations?

      Philosophy can be made out of anything. Or less -- Jerry A. Fodor

Re: Benchmark Arena Proposal
by PsychoSpunk (Hermit) on Jan 30, 2001 at 03:01 UTC
    This is quite a great idea. I had initially thought that regardless of the contents of this post that this would be a great idea, since I had envisioned a sort of deathmatch style of coding, with benchmarks being the ultimate deciding factor. Then of course, your idea brings a whole new realm to what I expected. It appears to be oriented to a Benchmark arsenal, with which you could deathmatch pieces of code and be very pleased with the results.

    ++ MeowChow.

    ALL HAIL BRAK!!!

Re: Benchmark Arena Proposal
by sierrathedog04 (Hermit) on Jan 30, 2001 at 21:22 UTC
    Your idea is a good one, but I have a question.

    Might some of the benchmarks you speak of be very dependent upon the type of operating system used, and how perl was compiled?

    If someone's perl was compiled very efficiently on an efficient operating system, might some performance penalties disappear?

      I think something like the machine specs, OS, and perl -V would be a prerequisite if the benchmarks are to be considered useful.

      Even better is having all the various WTDI being benchmarked on the same machine - that way we're only looking at Perl, not what different OS's strengths/weaknesses are.

      Pete

      I am mostly suggesting Perl vs. Perl cmpthese benchmarks, which tell you how one WTDI performs compares to another, usually regardless of what platform you are running on.

      The idea is that for most common tasks, operations, algorithms, etc., we will have a vast and open body of organized knowledge that usually provides:

      • A fast WTDI
      • An elegant and perlish WTDI
      • An obvious WTDI
      • A contorted and masochist WTDI
      • A just plain wrong WTDI

      Of course, these are not always mutually exclusive :)

Re: Benchmark Arena Proposal
by salvadors (Pilgrim) on Jan 30, 2001 at 21:23 UTC

    A good place to practice your optimising is Doug Bagley's Great Code Shootout, where he compares a variety of different scripting languages doing different tasks.

    It's quite fun.

    Tony

      Thanks for the link. I checked out how Perl did, and was somewhat shocked at what I saw.

      Perl ROCKS at text processing. It beat all other languages, including gcc, when it came to regular expression matching, and was near the top in spell checking, word concatenation, etc.

      On everything else Perl was, well, in the toilet. Indeed, the only language it consistently beat was tcl. When it came to method invocation, perl was the worst language tested in terms of CPU time.

      One point to consider is that these tests probably included the time for perl to first compile the test script and then run it. In some environments (mod_perl?) I believe that Perl will cache a compiled version of whatever it is doing, so performance would improve.

      Even so, what I am seeing is that for tasks in which the primary processing load is something other than text processing Perl may not be the most efficient execution language.

      That does not mean we should not use it. After all, assembler is the most efficient language that there is, but few recommend developing in it. However, I now believe that a competent programmer should excel in more than just perl. Like Larry. He knows Perl backwards and forwards, but he also knows C, and that is what he chose to write Perl in.

        I believe that Larry wrote Perl in C due to Perl's widespread use in the Unix environment, considering Unix was also written in C. He pretty much compiled various Unix language uses, such as awk, c... and created Perl. I would hope that he knows Perl backwards and forwards considering he did write it. Although, I must admit, I prefer coding in Perl because it's just so damn easy :) - thus probably why scripting in Perl is so popular. Of course scripting could be written in C if one were ever so inclined to do so, but as you stated above: "Perl ROCKS at text processing."!!! (exclamation points added by me!) And mod_perl is all that you say it is - faster - because it's a preloaded program. :)
      A very interesting link indeed! While Perl might not look to good in many individual tests, the scorecard page on that site allows assignment of weights to the benchmarks and under the default weights Perl does fairly well (I'll have to go back when I have more active brain cells and refine the weights a bit).

      Now that other languages have been raised in this thread, I'd like to add the point that (for me at least) one major benefit of Perl is greatly reduced development time - in the dark ages I did all my text processing in Fortran or C (funny, they don't include Fortran on that site ;-). Now I imagine this would be very hard to do, but has anyone ever attempted to compare the development times? Sure it would be hard to avoid just comparing the coding proficiency of people, and stats on LoC per day I've seen have usually been pretty indefensible. How about errors per day? Errors per line? Define a line or error! One might get around all these silly measures by simply comparing total time to meet an end-product specification, especially since many fields use (or are moving to) an end-product focus. Of course that ignores maintainability...

      All the same, I imagine someone (in some hairy-titled ACM journal perhaps) has looked at this question, and since it's an issue that Perl responds to well, it might be worth further investigation, vis-a-vis Perl advocacy.

      In any case, while I do find Perl vs. Perl benchmarks most amusing, all a benchmark will tell you is how fast a given platform can run the benchmark!

      --
      I'd like to be able to assign to an luser

        In any case, while I do find Perl vs. Perl benchmarks most amusing, all a benchmark will tell you is how fast a given platform can run the benchmark!

        I disagree. A good benchmark should *compare* 2 or more things, not to see how fast either runs, but to see how they run relative to each other.

        In this case Doug compares Perl to lots of other scripting languages - all on the same platform, and where possible, under the same conditions. The results are valid for seeing a lot more than how fast (in raw speed) something is.

        Tony

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: monkdiscuss [id://55103]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (10)
As of 2014-08-21 19:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (143 votes), past polls