|Perl: the Markov chain saw|
Benchmarking Your Codeby turnstep (Parson)
|on Apr 24, 2000 at 22:05 UTC||Need Help??|
Benchmarking your code
What is benchmarking?
Benchmarking is a way of measuring something - in this case, how fast your code runs. This is particularly useful when you are trying to compare two or more ways to do the same thing, and you want to see which one is faster. You are really measuring which way is more efficient for perl to do - the less work it takes perl, the faster it is.
Small differences add up. A slight change in a small section of your code may make a big difference, especially if that code has to perform a lot of work. For example, a different ways of sorting a collection of words may not matter much for 100 words, but if you have 100,000 words, the small differences start to matter. Also, it can be matter of style to make your code as efficient as possible. It is a good goal to aim for.
How to benchmark your code
Benchmarking is usually done with the (surprise) Benchmark module. This module is very standard, and is very likely already installed on your system. If not, grab it off of CPAN.
Benchmarking is not as simple as subtracting the results of one time call from another one - these are only accurate to one second, and are not a very good measure. The benchmark module uses the time function as well as the times function, which allows for a much finer measurement of milliseconds.
Here is a quick overview of the Benchmark module:
To use it, just type:
at the top of your code. Benchmark has three simple routines for you to use: timeit, timethis, and timethese. Each one needs to know what code to run (sent as the string CODE in the examples below), as well as how many times to loop through the code ($count in the examples below).
For a simple measurement of one piece of code, just use timeit. You will also need the timestr routine, which changes the times that Benchmark uses to a more useful string:
This can be a bit awkward, so Benchmark also has the timethis routine, which does the same thing as timeit, but also outputs the results. No timestr is needed this time:
The last routine is timethese, which is the most useful, as it allows you to compare 2 or more chunks of code at the same time. The syntax is as follows:
It returns an array, but this is often unused. Use of the 'alternative comma' is also recommended, to make it easier to read:
It will run each code in the list, and report the result with the label before it. See the example below for some sample output.
A final routine to know is timediff which simply computes the difference between two Benchmark objects:
The benchmark module has a few other features, but these are beyond this tutorial - if interested, check it out yourself: Benchmark.pm has embedded POD inside it.
For a simple example of benchmarking, let's compare two different ways of sorting a list of words. One way will use the cmp operator, and one will use the <=> operator. Which one is faster for a simple list of words? We will us benchmarking to find out. For this example, we will create a random list of 1000 words with 6 letters each. Then we'll sort the list both ways and compare the results. Here is our complete code:
Notice that we store the results of our sort into an unused variable, @temp, so that @words itself is never sorted, as we need to use it again.
Here is the result of running it with a count of 10:
The results tell us four numbers for each code. Notice that it also gave us a warning for the first one. This warning is only a guideline, but it is usually right - we need a higher count. Try to get the number of cpu seconds (the last number) to be at least 3 seconds or more for one of the measurements. In our example, let's try boosting the count to 150:
Much better! No warning, and some real times are generated. Let's look at each of the numbers. The first number is the elapsed time, or how many seconds the loops took by using the time function. This is not a very reliable number: as you can see, with 10 loops, one of the results was 0 seconds. Generally, you can ignore this one, except as a rough guideline. In particular, a reading of '0' or '1' is almost useless. Aim for at least an elapsed time of 5 seconds or more for the best results.
The next three numbers come from the function times, which returns much more detailed information. The first two numbers return the user and system time. Don't be surprised if the system time is often "0" or very low. These are not as important as the final value, the cpu time, which is what we are really interested in. This is the one you should use to make your comparisons. Try to get at least one of the numbers over 5 seconds - the higher the number, the more accurate your comparison will be. In this case, we can see that Method One, the <=> operator, is faster at 4.90 cpu seconds compared to the 7.13 seconds that cmp took.
Tips and Tricks
Here are some things to think about and watch out for: