Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re^4: "Automated" benchmarking

by salva (Monsignor)
on Jan 24, 2012 at 12:41 UTC ( #949672=note: print w/ replies, xml ) Need Help??


in reply to Re^3: "Automated" benchmarking
in thread "Automated" benchmarking

The overhead introduced by Devel::SmallProf for every Perl sentence executed is too high resulting in a non-linear distortion of time measurements.


Comment on Re^4: "Automated" benchmarking
Re^5: "Automated" benchmarking
by BrowserUk (Pope) on Jan 24, 2012 at 13:12 UTC
    a non-linear distortion of time measurements.

    I'd be interested to know more about the "non-linearity" if you have the time to explain further.

    But for the most part, the absolute accuracy of the timings is of little importance to profiling.

    The important factor is the relative timing, and the consistency those relative timings from run to run.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

      Say you have two Perl sentences A and B. A runs in 10us and is executed 20 times. B runs in 100us and is executed 10 times. Now suppose that the overhead introduced by Devel::SmallProf is 100us per sentence.

      Then:

      real_time(A) = 20 * 20 = 400us real_time(B) = 100 * 10 = 1000us measured_time(A) = ( 20 + 100) * 20 = 2400us measured_time(B) = (100 + 100) * 10 = 2000us

      So, you should be optimizing B, but Devel::SmallProf tells you to look at A.

      The current implementation of Devel::SmallProf performs a sub call, a call to caller, a hash lookup, a couple of tests and conditionals and two calls to Time::HiRes::time() for every sentence making the timings completely unreliable.

      I tried to reduce that overhead as much as possible on Devel::FastProf writing the DB sub in C/XS. But on Devel::NYTProf, they went further (IIRC) replacing Perl OP loop by their own and eliminating much of the overhead of the DB interface.

        Thank you for the concise and clear explanation. But ... :)

        Isn't the 'nulltime' compensation code meant to correct for (most of) that?

        # "Null time" compensation code $nulltime = 0; for (1..100) { my($u,$s,$cu,$cs) = times; $cstart = $u+$s+$cu+$cs; $start = Time::HiRes::time; &$testDB; ($u,$s,$cu,$cs) = times; $cdone = $u+$s+$cu+$cs; $done = Time::HiRes::time; $diff = $done - $start; $nulltime += $diff; } $nulltime /= 100; ... my($delta); $delta = $done - $start; $delta = ($delta > $nulltime) ? $delta - $nulltime : 0; ...

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?

      ++

      Very true.   “The presence of the experiment will affect the outcome.”   But if you can, in fact, presume that it does so in exactly the same way, the various test results will be comparable to each other, even though they will not match performance outside of the profiler.

      That can, however, be a big “if.”   Many of the things which affect the perceived speed of an application are external to it; they are environmental.   Only things such as these tend to produce a human-perceptible difference in speed for garden-variety applications (vs. say, the much more intensive ones that BrowserUK commonly deals with at his place of business).   So you might be testing the virtual-storage system ... how fast your disks can thrash (if they are overloaded), or how long it takes to page-in a very large working set (after which performance probably will stabilize and speed-up considerably, but if you stop the test too soon you won’t know it).   (The way I heard it said, by an actual biologist, was thus:   “be sure that you are actually measuring the mice, not the temperature of the room or the color of your shirt.”)

      In addition, actual pragmatic performance is very dependent upon the pattern of inputs, which leads me often to find that macroeconomics produces more useable intelligence than microeconomics.   Simply observing and logging the run time of “a complete job,” whatever that is, might be long-term more useful, and considerably easier.   Unless you truly suspect that the performance hog is located in “a particular function or set of functions,” and can be measurably confined to it, it might not be terribly informative to profile on a function level.   I mostly use profiling to discover “hot spots” in unfamiliar code.   Then, I look for ways to apply the advice in The Elements of Programming Style:   “don’t diddle code to make it faster ... find a better algorithm.”

        “don’t diddle code to make it faster ... find a better algorithm.”

        This is another of these fatuous "wisdoms" you are so ready to trot out.

        How do you think better algorithms get developed?

        Because thank goodness, there are some programmers that expend time and effort "diddling with code" to work out what works and what doesn't.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://949672]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (12)
As of 2014-10-01 16:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    What is your favourite meta-syntactic variable name?














    Results (29 votes), past polls