in reply to No More Meaningless Benchmarks!

Mark Twain said "There are three types of lies; lies, damn lies, and statistics."

Unfortunately the world is dumbed-down enough that people can easily be bamboozled by stats. Just reflect on advertising slogans and claims (a few contrived examples):

- "50% more vitamins than..." (Yes, but if X is Coke, and Y is Coke with a droplet of lemon juice in it, is 50% more actually significant? No.
- "Your risk of contracting (name your disease) is ten times greater if you eat pistachios." (Yes, but if the risk without pistachios is one in 50,000,000, who cares if your modified risk is 1:5,000,000? You're still more likely to die of a car wreck.)
- "Indexing a hash is twice as slow as indexing an array." Yes, but both are O(1) operations, which means if you're trying to optimize you might just want to look at the greater algorithm in which you're performing these lookups. If you actually care about the time differential, you might need to ask yourself why, and how you might eliminate the question in the first place.

What people often don't get is the concept of statistical significance. ...and relative significance (which is related). Teaching that is beyond the scope of this node. ;)

Dave

In Section
Meditations

Comment onRe: No More Meaningless Benchmarks!