Why, not How

[Ovid]

So there I was, reading this article entitled Why software still sucks and I started thinking about this issue. Let’s face it, software does suck. Why the subject of the aforementioned article seems unsure as to the reasons, I think I can come up with some:

With increases in power of hardware, companies are constantly trying to develop software that takes advantage of the potential. Unfortunately, the more complex something is, the more likely it is to have defects.

Market pressures also play into this. I don’t see a way around this. If company ‘M’ decides to rush a broken office suite to market and captures 90% of the market, company ‘Other’ can come out six months later with a much better market suite, but everyone’s trained on the broken model. Many would feel that if the first office suite is not too broken, it’s cheaper to work around problems than to retrain everyone.

Educated programmers are becoming increasingly more rare. How many times have you seen a “Programmer wanted” ad with a line like “Bachelor’s Degree in computer science or equivalent experience”? Everytime I see that I think "danger Will Robinson", but then I'm grateful because I don't have a BA.

Why does tilly consistently come up with slick algorithms to deal with problems? Education and experience. Why does merlyn routinely spit out nifty one-liners that make Perl look like the Yoga master of programming languages? Education and experience.

Many new programmers discover that they are very gifted with programming. Let’s face it, we’re handed seemingly Herculean tasks on a regular basis and somehow, some way, we manage to clean those stables. Some of us use shovels and others divert a river, but we get the task done. But at what cost? Do you know why binary trees are less desirable than B-Trees? Would you write a bubble sort when a quicksort or binary sort would be better? If you don’t have education or gobs of experience, how would you even know what the issues are? Personally, I only have an Associate of Science degreed. Many programmers that I know only have a high school education (and at least one is a high school dropout).

I’m becoming convinced that one of the biggest obstacles that we face is poorly-educated programmers. So many people try to reinvent the wheel not for educational purposes (I’ve written CGI parsing routines just to better understand what’s going on), but because we feel that we can do better. But do we? Find me a hand-written CGI parsing routine that works. 99% of them out there are broken and I’ll point out the flaws in them in a heartbeat. But that’s because I’ve educated myself in this one narrow area. Unfortunately, there’s too much for me to learn. The Renaissance man is dead.

You can’t control market pressures. Most don’t have the power to stop a company from overreaching themselves when trying to exploit hardware potential. However, you can educate yourself. That’s one way we can help improve on much of the dreck that gets shrink-wrapped.

I was reading through the Black Diamond thread and have mixed emotions about it. I would change one of the main statements to “You must be this tall to use Perl to its fullest advantage.” I think this is particularly true of those who reinvent the wheel without understanding the problem, or those who use cargo-cult code.

Don’t stop learning the abstracts. You can learn everything about creating TK apps, the HTTP protocols and how different sort operators in Perl work, but unless you keep abreast of the “why” in addition to the “how”, you’re going to be just another hack. And I mean “hack” in the journalistic sense, not in the 7337 sense. Frankly, I can’t claim to be particularly educated in this area. I have about one year of structured program design methodology pounded into my head in college and that’s about it. I started writing programs just knowing that I’m the most brilliant programmer around. That, of course, is bull$#|t.

Next time you're going to rail against monks who say "don't reinvent", "cargo-cult warning", or "I wouldn't use a regex" (gasp, sob) ask them monk why they would approach differently. Even if you don't agree, you can gain great insight into how problems are approached. Personally, I don't see any easy way around the general degradation of commercial software that I see, but poorly educated programmers is probably the easiest problem to identify. Don't be one of them.

Thanks for letting me spew.

Cheers,
Ovid

PS: Ironically, as a kid I knew that I didn't want to be a doctor or lawyer because then I would have to study for the rest of my life :)

Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.

[tye]

I feel that much programming should be turned into a real "engineering" discipline, requiring a license that ensures that you got an education from an accredited institution, keep up your education, and take professional responsibility for the code that you produce.

Then when your boss tries to apply market pressures to you, you can honestly say "Sorry, I can't let this product be released because it would be unethical and I could lose my license."

I'm not saying that coding be illegal without a license, but that many or perhaps all commercial products require at least supervision and approval of a licensed coder before money can be charged for them.

But this will never happen. A good alternative would be to have a voluntary system where you get a "Good Programming Seal of Approval" on software that meets some strict standards. Then many big groups will require this seal on important code such as the flight controls of commercial jet liners and things can snow-ball from there.

A common response when I propose this is that the cost and time of producing software would skyrocket and destroy the economy.

My response to that is that in my decade of professional software development, I've spent more time fixing problems caused by other peoples' code than I have writing my own code. If the code I got from other companies was of reasonable quality, I'd have much more time and could spend that making my software of reasonable quality as well.

So I think this idea would produce an initial slow down but in the end would make the software industry more productive (as well as making software suck a lot less).

Last week I mentioned that to my new manager and he said that this is actually proposed in a well respected book on software development. I'll throw the title in here when I find it.

        - tye (but my friends call me "Tye")

[jptxs]

I go from shop to shop to shop and I have seen in many of the manufacturing and engineering oriented shops that the management is well aware of how shoddy everything is on the IT side. The people who are used to the disciplines of engineering professions know that software - and hardware for that matter - should be regulated in the same ways, but also realise that until it is it's open season. Imagine having the ability to produce barely functioning product and have only caviat emptor to answer to? It's free money. No one is going to walk away from it yet.

The few instances where certian standards are required (like Oracle requires a trained Oracle Parallel Server professional to be attached to your site if you expect gold support for OPS), the professionals are just as bad as the managers. These people know they have companies over a barrel. There is a shortage of people who ahve the cert.'s and a need to have them there no matter what the cost. Again, free money.

The only resolve will be time and failure. As more and more people of the Gen X move up the socio-economic tree and more politicians become aware of technology issues as platform - which will come with the further penetration of technology into daily life of non-tech professionals - there will be demands for quality above quantity - and above new versions. Only then will we have a way to start regulating in some constructive way.

I think a VERY good question is what would such regulation do to the Open Source movement? Who's going to stake their certification on patches they didn't write? Interesting...

"A man's maturity -- consists in having found again the seriousness on
+e
had as a child, at play."                                   --Nietzsch
+e

[coreolyn]

Ya like the MSCE means something. Or how bout BS degree's that means they studied on outdated technology. I guess I just have a real deep rejection of this approach. If computers had been around before me and this type of system was in place, I may never have been able to qualify to work in this field. Yet more often then not, I find I'm helping those who supposedly have expirience and education in this business. I also have serious concerns over who would have the authority to decide when to pull a license.

Personally I really like some of the T-Shits they were selling in Monterey that simply said "Got Source?".

When push comes to shove source code speaks louder than credentials.

coreolyn

[tye]

No, an MSCE doesn't mean anything. It is a marketing gimmick. Nor is a BS much like a license.

And, again, I'm not saying you should need a license in order to work.

And I'm not even saying that licensed programmers will be the best programmers. I'm saying that they will be required to be trained on the issues that always get pushed aside in software development like risk management and security. And they will be held accountable. There will always be very talented programmers who produce great code, with or without a license.

As to pulling a license, that would be handled as it has been handled in other engineering areas for years. It isn't a simple thing.

        - tye (but my friends call me "Tye")

[royalanjr]

I will agree that there is a lot of software out there with all kinds of things wrong. But I do not think that you should need a license to be a paid programmer.

Say I got hired as a Perl programmer tomorrow.
I would be unemployed in two weeks, tops.

Why?

I simply am not that good yet.
That would become obvious, and I would be (hopefully politely) asked to leave.

If management has their eyes open, and cares, then only programmers that knew what they were doing would be hired. But as we all know, management is often blind and/or doesn't care.

Roy Alan

[Albannach]

<sermon>

Should you need a licence to design a freeway bridge? An aeroplane? A skyscraper? A real-time train switching control system? Pacemaker firmware? Insulin pump?

I am a licenced professional engineer (yea, whoop dee doo, but I have had some experience with this) and everywhere I've worked in my field (i.e. not software!) there have been many unlicenced staff working under the supervision of a licenced engineer. Most of the time this is required by law, but sometimes it is simply required by the client, who wants some assurance of quality (and no, ISO 9000 ain't gonna give you a better bridge).

Granted, the safety argument is a no-brainer, so let's just consider costs. A poorly designed road may be safe enough, but it might not last very long, meaning higher maintenance costs and capital replacement costs, both of which can be estimated fairly accurately. What does a bug in a banking system cost? An inventory control system? Is it because these are harder to quantify (that is if anyone wants them quantified which I doubt) that few seem to care?

Like it or not, I think software will become regulated as a profession sooner rather than later, and it's up to all you who think of yourself as software professionals to see to it that standards are set, met and kept. Just as a clarification, licencing doesn't have to mean Big Brother, as most professions are self-regulated under powers granted by the government, but not controlled by government. Sure it takes away the sort of microchip cowboy image that many enjoy, but that can be reserved for your time off. When society is paying the tab, it eventually learns to get what it wants.

</sermon>

--
I'd like to be able to assign to an luser

[tye]

But I do not think that you should need a license to be a paid programmer.

Neither do I. I think that most companies need to be required to hire at least one licensed programmer. Most of the programmers will continue to be unlicensed and continue to be paid but will be supervised by licensed programmers.

Of course, there are no licensed programmers yet so even that can't happen overnight. I'd just like to see things get started in that direction because I'm sick of fixing other companies' code.

        - tye (but my friends call me "Tye")

[gharris]

I like the idea of the license if for no other reason than to be able to tell my boss:
"Sorry, I can't let this product be released because it would be unethical and I could lose my license."
On a more serious note, I think if there was some 'independent' organization, like "National Assocation of Computer Scientists and Software Engineers" or something along those lines that could manage the whole licensing thing like similar organizations do in all other aspects (M.E., electricians, etc..) it could help to avoid the MSCE phenomenon and maintain expectations beyond a piece of paper.

--Glenn

[zzspectrez]

Having a liscense really doesnt mean anything. I work in the medical field as a radiographer, taking 'xrays'. To get a job doing this in California requires a state license and usually also national license. To keep your license you are required to take so many educational units a year. However.. In my experience quite a few of the licensed profesionals either couldnt pass the test if they had to take them today without lots of reviewing or do not bother to follow some of the basic ideas that were the reason for implementing the license in the first place.

Realisticly, threat of loosing your license because of being unethical or because of poor job performance would not be effective. Unless of course were talking negligence resulting in something drastic. Like... hum.. Giving buggy code to a governement agency resulting in massive data loss! Otherwise nothing is going to happen.. Who is going to inforce it?! Think about it, even with lawsuits and deaths there are still doctors practicing medicine who shouldnt..

Do you really think it would help enusre better code??? Better education, yes... But just because you are dedicated enough to go through school or study for licensing tests doesnt mean you are a good programer. It will not bring responsibility for the code produced.

The responsibilty for code produced is shared between the programers who write the code and the people or business's that release and sell that code.

Thats my 2 cents.
zzSPECTREz




In theory it is a good idea... In reality, it wouldnt do any good I fear.

[mirod]

I mostly agree with you here, the more you learn, the more you look into the "why" of things and the more you're likely to be a good programmer.

I don't think "Software Sucks" though, and above all I don't think it is fair to compare programming to other kinds of engineering. At least not in terms of "hardware engineers deliver stuff that works while software wankers can only spit out pathethic pieces of crap".

I would roughly divide software in 2 kinds: complex and simple (brilliant isn't it?)

An OS, a data base, Perl itself, those pieces of software have a degree of complexity that's close to a plane, in terms of design, so they should be compared to planes, or satellites. And let me tell you, I have been involved in planes and satellites, and they are pieces of crap! They are full of bugs! Overall the system sort of works, and there is enough redundancy and the margins of error in each sub-system are such that it rarely crashes. But no plane flies with all systems 100% OK. It just doesn't happen. So why hold software to a higher standard? Perl works, Hell, even Windows is not that bad (as long as it's not used in planes)!

And as for simple software, the business rules are such that it is more a craft, where a skilled (or not-so-skilled) craftman delivers a product, than anything close to hardware engineering, where you have the time to thouroughly test a design before going into production. If that kind of software is full of bugs it's just because the market has judged that it made sense economically.

So in short the business model of software is different from the model of other kinds of engineering so I don't think it's fair to compare them.

And let's drop the "Software Sucks" thingie.

Not to mention the usual "if the hardware is faulty we'll fix it in the software".... we can't do that ;--(

[Petruchio]

Before I address Ovid's points, I'd like to touch on a few of Jaron Lanier's. Indulge me. The article annoyed me.

First off, software rocks.

Hardware can point to a relatively sunny track record because it's relatively simple. Stunningly complex, yes, measured along a certain dimension, but far, far simpler than software. Software encompasses operating systems, communication protocols, programming languages, hardware definition languages, text interfaces, graphical interfaces, browsers, games, Bob the Annoying Paperclip, you name it. Hardware provides the canvas. This is as it should be. We figured out a long time ago that it was most efficient to design a line of watches with identical hardware, and differentiate them with software. Over the years, a larger and larger share of the complexity we're dealing with has been pushed onto the software side.

Further, whenever there's a sense of urgency in an undertaking, gaining the ability to do something is always going to precede the ability to do it really well. Our core software (like Unix and Perl) is darned good, and our less solid stuff can do a heck of a lot.

Well, that is, Unix is darned good if you rate it as a tool which has been useful in conquering vast complexity. But if you're very much used to the way people may cater to you, it's easy to be miffed when the world doesn't. I have little admiration for the fatuous complaints of people who expect the cutting edge to bend over backwards for them, or who seem to regard the ascent of language over cave-drawings to have been mankind's biggest setback.

Ok, that's out of my system. Ovid's remarks are much more agreeable. :-)

I see very few companies concerning themselves with exploiting the potential of hardware. The mere fact that "Java" ever became a buzz-word is, I think, an indicator that the general drift of things is quite different indeed.

I think that educated programmers are becomming more common, but that programmers in general are becoming more common at a much faster pace. This distinction notwithstanding, better education is needed. As I see it, programmers generally start out in one of two ways: they learn the theory, and then how to get things done, or they learn how to get things done, and then learn the theory. Hopefully. The second type (of which I am one) is becoming more common at a faster rate than the former, as the lucrative wages draw people in from other fields... and the danger with these folks is that they will never find their way to learning the theory.

So yes, quite right, Ovid. But it's noteworthy that nobody seems to be trying very hard to teach the theory either. Plenty of books and sites out there helping you master Perl or C++ in a half an hour, but (with a few notable exceptions) virtually no tutorials on algorithms or data structures. The assumption seems to be that such things are interest only to serious programmers and computer scientists, and so your average English-major-turned-programmer muddles about in ad hoc scripts, ignorant or despairing of anything better. Maybe someone around here should help fill the gap.

Anyway, them's my $0.02. One good spew deserves another. :-)

[coreolyn]

Great spew actually!

I don't even have an associates, nor have I even quarter of the skillset that you and the others mentioned in your post have. But since before the PC first hit the market I've had to brutally expirience OS and language evolution. No matter what OS or language the basic problems and resolutions have been essentially the same. Get the data from here to there without blowing up the program or crashing the system -- faster, better.

After 20 years it's finally starting to finacially pay off, but as I'm twice as old as most of my co-workers I definately would finish college had I to do it again.

But I'm off on a child process, back in the main of your spew, you claim that The Renaissance man is dead. IMHO I think we are just now entering a period where this may soon change. Looking around the work place I find highly qualified super-specialist that have to spend most of there time coordinating and meeting with other specialists so they can get back to their specialty and actually accomplish something. Eventually the inneficiency will be understood by the wonderfull management specialists of the world.

Speaking of which (management types), these folks have more of an influence on "Why software sucks" than you may be realizing. The 're-invention of the wheel' syndrome usually comes from the rewarding of new novel solutions that have all the right buzzwords. It is easier for management types to understand 'buying' new technology than to consider how it integrates into what they've already invested in. How could they? There aren't specialist in that!

You are very right though, The answer to a 'why' over a 'how' usually yields reusable thought components. It's more mentally OO. and the resultant understanding can usually be inherited into new and novel problems that may not even have anything to do with programming.

coreolyn

[Elgon]

Wow! Heavy thoughs and thought provoking indeed.

I am a beginnner at Perl and in some ways, at programming too, in that I re-took it up about 6 months ago after a leave-of-absence for 7 years or so. What excites me about programming is that is something which is mentally stimulating, requires problem solving abilites and abstract thought. I will never be a truly kickass programmer: why I know this I cannot say but I feel it when I got to church, when I... mental note to self: give the matrix a rest for a while

Back to seriousness. I feel that in many ways the whole problem is one of a general education in maths and logic. I see this regularly at University (Chemistry major, French minor): People who get astounding marks in exams but can't understand the reasoning behind what they regurgitate, others who think that we study at University to get a degree or learn about a given subject. These people are not stupid, they merely lack experience: I do too in the programming field. Can this be part of the problem? Do many of these coders who are churned out have the vision to take their knowlege and extend it through reasoning and logic (and when all else fails, by trial-and-error) to cover methods and areas their course hasn't covered in detail?

The key to programming is not merely having an encyclopaedic knowlege of a given language, nor is it even understanding all aspects of a language, I think that being able to follow the consequences of your actions deeply and logically is the most important aspect. That is: be able to mentally map out the various probable paths of your program when the variables have the correct values, when they don't have the correct values and then to notice the boundary conditions and udjust accordingly. looking back at that sentence, I'm not sure if it is very deep or just crap

Experience is the key, combined with the ability to wing it when really necessary!

By reading this post you have agreed to abide by the following license conditions: 1. No flaming Elgon 'cause you think he's full of sh*t. 2. Waiving all rights to legal redress after self-inflicted injury through apoplectic rage. Elgon

[toadi]

Well I don't have a license nor did I follow any education to be a programmer.
I only started working as a programmer in february of this year. I joined perl monks a bit after that. Wel I know most of the theoretical stuff that need to be known. The more I program the better I get and maybe when I have 20 years off experience I can be a 'guru'(this is a very big maybe).

I went from programming websites(HTML, Javascript) to programming apps for the web in PHP and now I use perl on a regular basis. I build apps to interact with oracle or mysql. On the side a administrate several linux webservers.

So you see I came al long way in the time-frame of 1 year.

Now to widen the perspective: it's not because you get educated at school for programming makes you have the right spirit for it. You need to read a lot: books on the web. Also interact with fellow collegues. And realise your code is not always the best solution. And above all try and program and have *FUN*. Yes I have fun programming, you guys make it all sound so serious. I'm also serious doing my job good and program well, but I need to have fun doing it.

Don't you ?

--
My opinions may have changed,
but not the fact that I am right