KISS and Occam's Razor

[hsmyers]

Lately there have been at least two nodes (If it's not broken, don't fix it and Design Patterns Considered Harmful) where the basic notion could be reduced down to the ever familiar acronym KISS. Which in turn got me to thinking about the origins of the idea of simplicity as a shield from complexity, which in turn reminded me of William of Ockham. Sort of a connections thing. To explain a bit:

KISS

“Keep it Simple Stupid”

William of Ockham

Frustra fit per plura quod potest fieri per pauciora or “It is futile to do with more what can be done with less”

While it is hard to avoid the KISS principle, not everyone has heard of brother Ockham, let alone his famous razor.

Briefly William of Ockham was a 14th century (c. 1285-c.1349) logician and Franciscan friar. Ockham was the village in the English county of Surrey where he was born. For those interested in alternate spellings, Occam is the latinized version of Ockham. One of the more influential philosophers of his time, his ideas (referred to as Occam's Razor) are still quoted today:

• “Pluralitas non est ponenda sine neccesitate.” which translates to “plurality should not be posited without necessity.”
• “non sunt multiplicanda entia praeter necessitatem.” or “entities should not be multiplied unnecessarily.”
• “Quia frustra fit per plura quod potest fieri per pauciora.” “For it is pointless to do with more what can be done with less.”

More than 600 years later, he is still influential:

• What can be done with fewer [assumptions] is done in vain with more—Ernest A. Moody
• Everything should be made as simple as possible, but not simpler—Albert Einstein
• The research worker, in his effort to express the fundamental laws of Nature in mathematical form should strive mainly for mathematical beauty. It often happens that the requirements of simplicity and beauty are the same, but where they clash the latter must take precedence—Paul Dirac

For those who want to read more, here are some URLs:

• http://www.uni-rostock.de/fakult/philfak/fkw/iph/strobach/veranst/mittelalter/OCKAMLAT.html
• http://www3.oup.co.uk/mind/contents/
• http://www.google.com/search?q=cache:qi5Qdua8Ssc:pespmc1.vub.ac.be/OCCAMRAZ.html+Occam%27s+Razor&hl=en
• http://www.medinfo.ufl.edu/year1/bcs96/interv/occam.html
• http://skepdic.com/occam.html
• http://www.cco.caltech.edu/~maronj/text/razor.html
• http://www.nd.edu/Departments/Maritain/etext/scholas5.htm
• http://home.xnet.com/~blatura/skep_1.html
• http://www.theness.com/newsletter.html
• http://www.weburbia.com/physics/occam.html

Note: Some of you may wonder why I didn't list his most famous quote: “non sunt multiplicanda entia praeter necessitatem” or “entities should not be multiplied unnecessarily.” Easiest answer is that I'm not entirely sure that he actually said it. I've run across at least one reference that suggests otherwise. I haven't been able to track it down so I don't know for certain. Any way I just managed to sneak it in as a note, so no problem!

Update: Fixed bracket typo.

–hsm

"Never try to teach a pig to sing…it wastes your time and it annoys the pig."

[tachyon]

Along similar lines, one of the more interesting things I have observed is that the less experienced a practitioner of any given art, the more likely they are to look for the rare and unusual explanations *first*. One point I constantly impress upon my students is "Common things occur commonly" often more amusingly surmised as "When you hear hoofbeats think horses not zebras".

The common, run of the mill, and obvious answer is usually the correct one. It's always fun when you have to track down a zebra though.....

Thanks for an amusing meditation.

cheers

tachyon

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

[Dominus]

Says tachyon:

the less experienced a practitioner of any given art, the more likely they are to look for the rare and unusual explanations *first*.

That seems obvious. Why are inexperienced practitioners more likely to look for unusual explanations? Because they don't know yet what the usual explanations are, of course.

It seems to me that one of the most important forms of experience that people get is to learn which problems are common and which ones are rare. Sometimes beginners come to me with broken programs and ask what the problem is and I find it instantly. But when they ask me how they could have found the same problem themselves without asking for help, I don't have very good advice: I looked for that problem because I knew it was a common problem, and I knew it was a common problem because I had made the same mistake six times myself.

I agree with your observation that beginners often look in the wrong places. But I don't think that advising them to look in the right places is very helpful. If they knew what the right place was, they wouldn't be beginners.

--
Mark Dominus
Perl Paraphernalia

[tachyon]

Of course what you say is correct that as you get more experienced you know where to look. Not only that you learn *how* to look. However even programming beginners should have learnt to look for say:

• typos
• missing semicolons
• mismatched quotes
• incorrectly paired curlies

as these are probably the 4 most common errors. Ever had someone with a simple typo suggest some bizare theorem for the disfunctionality of their code? That was not really the point. It was simply an idle refletion on life the universe and everything.....and I always like the thought of hoofbeats and zebras....

Perhaps the analogy is better with what I teach which is Emergency Medicine rather than coding.

compliments of the season to you

cheers

tachyon

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

[snafu]

Very nicely stated. And so far this has been an enlightening meditation. Now, to go a bit further to your thoughts I ask; Isn't the teacher suppose to be a "right place" to search for the answer when the answer isn't obvious? :)

_ _ _ _ _ _ _ _ _ _
- Jim
Insert clever comment here...

[mstone]

For the sake of a nice segue, I'm going to commit something that looks a bit like a dictionary flame. Since I hold dictionary flamers on a par with spammers (somewhere below pond scum.. a mental image I find pleasing), personal ethics demand that I first post a disclaimer, and give props where due.

So: bravo.. that was a nice, scholarly overview of Occam's Razor. It's rare to see someone actually go back to the source and discuss the idea thoroughly. The list of links alone earned you my ++.

Now I'll grab my hyper-pedantic asshole hat long enough to note that 'simplicity' and 'minimalism' aren't always the same.

William was talking about minimalism.. not using two pieces where one will do. The word 'simplicity' arrived later, which is nicely illustrated in the series of equivalent quotes you listed. All three of the earlier sources boil down to counting entities, as does Moody's from the present day. Only when we get to Einstein and Dirac does the more general concept of 'simplicty' come into play.

Sure, there's a version of simplicity that equals minimalism, but there's also a concept of 'piecewise simplicity' that almost inevitably leads to multiplication of entities. To quote H.L. Mencken:

For every complex problem, there is a solution that is simple, neat, and wrong.

That's something we, as programmers, need to care about.

I can't count the number of times I've seen code that starts from a few simple, easily-stated assumptions, then evolves into a Rube Goldberg machine due to inconsistencies among the assumptions. I've also spent a fair amount of time trying to pry coders loose from those simple-but-inconsistent assumptions so they can go back and write something that actually works. Some have defended themselves with the KISS principle, because digging out inconsistencies can be hard. More than once have I had to explain, "simple is not always easy," and, "simple means everywhere.. not just right here."

In the language of systems analysis, it's called 'globally suboptimal local optimization'.. fixing a problem in one place by creating an even worse problem somewhere else. If you really want to see that principle in action, divide a project among half a dozen coders, and reward them based on whose code profiles as fastest. It's a rare to find team that doesn't have at least one person willing to shove work off into somebody else's module.

So.. simplicity is a great goal, but remember to watch the global perspective. There are few better ways to write absolute crap than to get perky about making things 'simple' at the local level.