XP is just a number PerlMonks

### Re^5: "Automated" benchmarking

by BrowserUk (Pope)
 on Jan 24, 2012 at 13:12 UTC ( #949678=note: print w/replies, xml ) Need Help??

in reply to Re^4: "Automated" benchmarking

a non-linear distortion of time measurements.

I'd be interested to know more about the "non-linearity" if you have the time to explain further.

But for the most part, the absolute accuracy of the timings is of little importance to profiling.

The important factor is the relative timing, and the consistency those relative timings from run to run.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

Replies are listed 'Best First'.
Re^6: "Automated" benchmarking
by salva (Abbot) on Jan 24, 2012 at 13:42 UTC
Say you have two Perl sentences A and B. A runs in 10us and is executed 20 times. B runs in 100us and is executed 10 times. Now suppose that the overhead introduced by Devel::SmallProf is 100us per sentence.

Then:

real_time(A) = 20 * 20 = 400us real_time(B) = 100 * 10 = 1000us measured_time(A) = ( 20 + 100) * 20 = 2400us measured_time(B) = (100 + 100) * 10 = 2000us

So, you should be optimizing B, but Devel::SmallProf tells you to look at A.

The current implementation of Devel::SmallProf performs a sub call, a call to caller, a hash lookup, a couple of tests and conditionals and two calls to Time::HiRes::time() for every sentence making the timings completely unreliable.

I tried to reduce that overhead as much as possible on Devel::FastProf writing the DB sub in C/XS. But on Devel::NYTProf, they went further (IIRC) replacing Perl OP loop by their own and eliminating much of the overhead of the DB interface.

Thank you for the concise and clear explanation. But ... :)

Isn't the 'nulltime' compensation code meant to correct for (most of) that?

# "Null time" compensation code \$nulltime = 0; for (1..100) { my(\$u,\$s,\$cu,\$cs) = times; \$cstart = \$u+\$s+\$cu+\$cs; \$start = Time::HiRes::time; &\$testDB; (\$u,\$s,\$cu,\$cs) = times; \$cdone = \$u+\$s+\$cu+\$cs; \$done = Time::HiRes::time; \$diff = \$done - \$start; \$nulltime += \$diff; } \$nulltime /= 100; ... my(\$delta); \$delta = \$done - \$start; \$delta = (\$delta > \$nulltime) ? \$delta - \$nulltime : 0; ...

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

A nulltime compensation is effective when the overhead is relatively small compared to the sentence execution time. But when it is much bigger as in this case, the compensated overhead (aka error) can still be of the same order of magnitude of the time we want to measure.

Guess why the last line you have quoted from the source is there!

package DB; use Time::HiRes; my %testDB = ( null => sub { my(\$pkg,\$filename,\$line) = caller; \$profile || return; %packages && !\$packages{\$pkg} && return; }, work => sub { \$do_something{foo} = 'bar'; my(\$pkg,\$filename,\$line) = caller; \$profile || return; %packages && !\$packages{\$pkg} && return; }); my \$n = 10; for (1..\$n) { for my \$t (keys %testDB) { my \$testDB = \$testDB{\$t}; my \$nulltime = 0; for (1..100) { my(\$u,\$s,\$cu,\$cs) = times; \$cstart = \$u+\$s+\$cu+\$cs; \$start = Time::HiRes::time; &\$testDB; (\$u,\$s,\$cu,\$cs) = times; \$cdone = \$u+\$s+\$cu+\$cs; \$done = Time::HiRes::time; \$diff = \$done - \$start; \$nulltime += \$diff; } \$time{\$t} = \$nulltime / 100 * 1e6; } printf "null: %.2f, work: %.2f, delta: %.2f\n", \$time{null}, \$time{work}, \$time{work} - \$time{null}; }
Re^6: "Automated" benchmarking
by sundialsvc4 (Abbot) on Jan 24, 2012 at 14:19 UTC

++

Very true.   “The presence of the experiment will affect the outcome.”   But if you can, in fact, presume that it does so in exactly the same way, the various test results will be comparable to each other, even though they will not match performance outside of the profiler.

That can, however, be a big “if.”   Many of the things which affect the perceived speed of an application are external to it; they are environmental.   Only things such as these tend to produce a human-perceptible difference in speed for garden-variety applications (vs. say, the much more intensive ones that BrowserUK commonly deals with at his place of business).   So you might be testing the virtual-storage system ... how fast your disks can thrash (if they are overloaded), or how long it takes to page-in a very large working set (after which performance probably will stabilize and speed-up considerably, but if you stop the test too soon you won’t know it).   (The way I heard it said, by an actual biologist, was thus:   “be sure that you are actually measuring the mice, not the temperature of the room or the color of your shirt.”)

In addition, actual pragmatic performance is very dependent upon the pattern of inputs, which leads me often to find that macroeconomics produces more useable intelligence than microeconomics.   Simply observing and logging the run time of “a complete job,” whatever that is, might be long-term more useful, and considerably easier.   Unless you truly suspect that the performance hog is located in “a particular function or set of functions,” and can be measurably confined to it, it might not be terribly informative to profile on a function level.   I mostly use profiling to discover “hot spots” in unfamiliar code.   Then, I look for ways to apply the advice in The Elements of Programming Style:   “don’t diddle code to make it faster ... find a better algorithm.”

“don’t diddle code to make it faster ... find a better algorithm.”

This is another of these fatuous "wisdoms" you are so ready to trot out.

How do you think better algorithms get developed?

Because thank goodness, there are some programmers that expend time and effort "diddling with code" to work out what works and what doesn't.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

Create A New User
Node Status?
node history
Node Type: note [id://949678]
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (4)
As of 2018-05-25 11:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
Voting Booth?
World peace can best be achieved by:

Results (184 votes). Check out past polls.

Notices?