Code re-use: productivity gains vs. skill deprecation

[Phemur]

I've been writing software for a relatively short period of time. During that period, I've been striving to increase my productivity and code quality by reading as much as I can, and coding as much as I can. I'm now faced with a bit of a paradox.

A fundamental principal of coding (which is applied to almost any other discipline) is "why re-invent the wheel?". In other words, why re-write a piece of code that's already been writen and tested, and which is available for free? By re-using this code, you avoid wasting time on designing, implementing, testing and debugging the component, thus increasing productivity and quality.

Another fundamental principal of engineering and science is "know your fundamentals". A prof of mine once told the class that 80% of engineering problems can be solved with first-year principles.

Here's the problem. If the most effective method of improving your coding skills is to code, and the most effective way to increase productivity and software quality is to re-use code (i.e. not write it yourself), then can you improve skill and maintain productivity? Is it possible to learn coding fundamentals without coding? Is general knowledge of an algorithm or data structure (such as a linked list) enough to write quality code, or does a developer need to know the implementation details of each algorithm and data structure?

Phemur

[Molt]

I personally tend to divide code I'm working on into either 'Productivity' or 'Experimental'.

When writing productivity I'll be doing my level best to employ code reuse to the hilt, I'll be avoiding the things I don't know too well and sticking as much as possible to a nice and easily-maintainable subset of my skillset. This is the bread-and-butter, pays-the-bills code. When doing this code I tend to concentrate on best practises, and improving my own processes. I use my knowledge of principles and the language, obviously, but I in no way attempt to keep them up to date or active at this stage.

Experimental is the other side of the coin, and is an entirely different proposition altogether. Attacking new technologies, or just trying to push back my boundaries by coding things I do know at a low level. Normally coding practises go out the window here, I still do the -w and use strict generally but I rarely document as the scripts probably won't get run again after today and anything worth having in them should be obvious. Limits are pushed now, I've done horrendous things here like coding my own Data::Dumper, my own CGI.pm, versions of most low-level data structures in entirely inappropriate ways, and other such perversions. This is where I learn, this is my playground

The good side of this approach is my productivity on bread-and-butter code is getting far better than it was, giving me the time to look into my own Data::Dumper, a quick knock-up distributed application, or horrific ways to spindle, fold, and mutilate closures. When I learn something worthwhile I make sure I know it well, and it's limitations better, and feel ready to use it in production.

The time you save by building foo for production code and so avoiding the development and testing of it means you can spent the time writing a quick'n'nasty foo for non-production code.. looser rules, and you still keep your skills up as here it is the principles that count.

A new way I'm beginning to look into the experimental phase is to get the tests for a standard module, and expand upon them with some of my own, and then code a module using the 'It must pass these tests' as my specification. Looks a promising approach to learning, and I may even spot areas where I may be able to eventually improve on the original.

[derby]

... the most effective method of improving your coding skills is to code

the most effective method of improving your coding skills is to inspect and understand well written code. This fits very nicely into re-use and extremely nicely into perl (CPAN) where inspecting the code is quite an easy thing to do from a licensing and data format (mostly text) point of view. (As compared to some proprietary vendor where all you get is the documentation and an object library).

It is possible to learn coding fundamentals without coding but it would be damn hard to implement a solution without coding. But ‹sarcasm›some CASE tool vendors say it can be done‹/sarcasm›.

As for knowing algorithms and data structures. You need to not only know them but more importantly know when to use which type. In theory, that means you don't have to know exactly how the bits and bytes are arranged. But in reality, I've seen stuff called one data structure but actually implemented as another (ditto with algortithms).

-derby

update: Inspecting badly written code can be just as helpful.

[perrin]

So are you saying that if not for the time you spend re-inventing CPAN modules you wouldn't have any code to write? I want your job! I find that even though I take pains to use CPAN modules whenever it's remotely possible, finishing all the features my users want would still take more code than I can write in my lifetime.

Also, one thing you're forgetting is that the popular CPAN modules are better than one person could hope to write. They are the product of thousands of people debugging, patching, documenting, supporting, etc. You're not just saving time; you're improving quality.

[Molt]

This is true, the quality of much of CPAN is quite remarkable in parts. I do see their point with regard to skills however, it's too easy to rely on the high-level view you end up with when using modules and end up not knowing the 'essentials' of computer science such as data structures etc. I don't think they was saying that they'd not have any code to write, more that they feared their command of the 'computer science' aspects would wither and die.

I've seen this reach a quite significantly scary level in some of the Java programmers I've worked with. They're used to their nicely insulative APIs, and that is good enough 'to get the job done' reasonably well most of the time, but when the problem suddenly needs more indepth knowedge of datastructures, network protocols, graphics algorithms, general optimisation, or other things they've been insulated from they just fell to pieces as they couldn't deal with it. It seemed anything outside their little books of APIs was outside their knowledge and abilities.

They almost all did computer science degrees, so assumedly were shown at least the data structures and networks parts, but now they've spent so long in their supportive language that they're unable to deal without it.

I'm not saying that code reuse is bad, far from it as can be seen from my post above, rather that reliance upon higher-level APIs/modules can be detrimental to your low-level understanding. Reuse, but make sure you still know what's going on.

[perrin]

I would say it's not very important that these Java guys suck at graphics algorithms if they are great at rapidly building web applications and that's what their line of work calls for. They may not be uber-coders, but they may have a lot of very satisfied users who appreciate their talents.

For those who long to earn more programming street-cred, there are always new areas where modules have not yet been written. Everyone knows something they think should be on CPAN that isn't there yet. Or else they know a feature that an existing module needs. Or a set of unit tests that an existing module needs. There are lots of ways to get experience without re-inventing wheels.

[dws]

If the most effective method of improving your coding skills is to code, and the most effective way to increase productivity and software quality is to re-use code (i.e. not write it yourself), then can you improve skill and maintain productivity?

On the surface, the answer is Yes. Assuming that you're not close to 100% reuse, you'll still be writing code, and there are always opportunities to increase coding skill.

Let's raise this a level and ask a meta question:

Given a choice between increasing your coding skills while holding productivity constant, and increasing your productivity while holding coding skill constant, which would you choose and why?

This meta question is important, because it gets at whether you're answering the right question.

I approach this with two beliefs: First, that there is a point of diminishing returns on the coding skill curve. There is a place on the curve where you the effort one needs to apply to get better is effort traded off against other opportunities. (It might take several years to reach that place^*.) Second, that coding skill is a means to and end, and not, tempting as it might be, an end to itself.

*See Peter Norvig's Teach Yourself Programming in Ten Years.

[Phemur]

That's a very interesting question you've asked. Here's my take on it.

I think the answer is "it depends". At the beginning of your programming career, coding skill is essential, because if you have no skill, then you can't produce. As your coding skills increase, so does your productivity.

Switching emphasis to productivity occurs once coding skills reach a level where, as you said, increased skill doesn't translate into increased productivity. And I think the real trick is determining what that level is. It's a combination of goal (for a given problem, how much skill is required, and when must the solution be available) and cost of improving your skills.

I partially disagree that coding skill is simply a means to an end. Some people aren't programmers by trade. They do so for the pure pleasure of coding, in which case productivity isn't part of the equation. For the purpose of this discussion, I'm talking about professional programming, in which case you're absolutely right.

Phemur

[FoxtrotUniform]

1. I don't know about you, but in my experience there's plenty of new code to be written. Most of it falls squarely in the category of "Joe code" -- I don't spend my days implementing constraint optimization systems or Fibonacci heaps (sigh); rather, I write data munging scripts for in-house tools. CPAN modules help, but they don't do all that I need.

   So even reusing as much code as I can, I still have the chance to write plenty of code, and most of it's low level glue code, dealing with strings, hashes, refs, and regexes: Perl fundamentals. My coding style gets a good work-out, my Perl-slinging stays in shape, and I get a warm fuzzy feeling from all the wonderful code reuse.

2. There's plenty of existing code that hasn't been fully tested or debugged, or that's architecturally flawed. "Don't reinvent the wheel" doesn't necessarily apply when the wheel's more of an ellipse.

3. I do a lot of coding on my own time, where "maintaining productivity" isn't as great a concern as maintaining interest in whatever I'm doing, which usually means that I should spend more time writing code than figuring out what design pattern to follow with the dozen or so modules I can get off of CPAN.

--
:wq

[brianarn]

I really like how Molt breaks it down into productivity vs. experimental, and have some thoughts akin to it.

Currently, I'm a CS student at the University of New Mexico, and as such I've reinvented a lot of wheels in my educational career, from simplistic C++ programs that take simple input such as 3d+2 and produce output such as 12, to implementing my own stacks/queues/lists, only to find that someone else had already created it (in the C++ world, the STL was an incredible thing to learn about). Had I not reinvented the wheel in these educational instances, rather than just used some prefab template, I feel like I wouldn't have gotten much out of the experience. Granted, when I was first shown some of the more advanced code available to me, I was frustrated to have had to go through my own creation process, but have since realized the true value I received from developing my own linked lists/trees/stacks/queues etc.

I'm also a systems programmer for my university, and as such I write perl scripts like there's no tomorrow, simple 'throw-away' scripts most of the time, sometimes large complex CGI apps. No matter what I'm writing, I try and reuse code anywhere I can get my hands on it, from simple things I'd written for myself as a playground to other past throw-aways. As such, even when I'm writing a simple script that I am told will only be ran once, warnings and strict are always a part of it. A friend of mine who is a Perl coder but non-monk and I get into fun arguments about throw-aways. He'll actually use $filecontents = `cat $filename` to read in a file for a fast throwaway - nothing irks me more. I'm just waiting for the day that he hands this throwaway to someone who blows it out into a giant app, and then some malicious user sets $filename to "; rm -rf /"

One area where I've been stuck is not having the ability to install CPAN modules (I'm actually forbidden from installing anything on a production machine) - so there are times when I have to do a little reinventing, but half of the time if it's a small module, I'll install it on my machine, open it up and see what they did, and potentially copy the code verbatim over into my script - it makes the script bigger, but I get tried and true code that works.

Reinventing pieces of code has great merits in an educational sense, and I think that any developer/sysadmin worth anything will probably rewrite many pieces of code in order to learn. Reading code is a wonderful way to learn good habits (so long as you take away the Good and leave the Bad), but I feel that actually going out and writing/debugging your own version is an excellent learning experience as well.

~Brian

[rinceWind]

I think that the first, excellent question asked has been very thought provoking, and has generated much discussion.

Is it possible to learn coding fundamentals without coding?

It is very important here to define terms here. You need to define exactly what you mean by coding. To answer the question as it stands, I would say "not usually". This is from looking at my own cognitive experiences learning new programming languages (OK, my head is full of syntaxes and techniques from other environments, but that's why people hire me). I would also suggest that there is a danger of cargo cult programming and all the evils that entails.

Is general knowledge of an algorithm or data structure (such as a linked list) enough to write quality code, or does a developer need to know the implementation details of each algorithm and data structure?

I would definitely say yes. This is what is meant in OO speak by encapsulation. This is a GOOD principle, which leads to loose coupling and high maintainability. What is important here is the interface definition. Before you can call another module, you must understand its interface, and be able to predict results from it without running any code.

[strat]

I think it depends on what you want to reach.

At first, the planning phase of a software is very important. And if you are able to split up the huge problem into small pieces you can solve easily in a very good and flexible way, very often you can solve problems with just very basic code.

If you want to improve your knowledge in a certain language, you need a lot of training, which may become difficult without doing a lot of coding. And without a lot of coding, you might never find out about many of perl's idiomatics.

When I write some code for customers, I often try to write it as easy as possible and only as complicated as absolutely necessary, because very often, these programs are enhanced or maintained by people who only have a basic perl knowledge or the like. Or if I write a code (e.g. at perlmonks or www.perl.de) which is addressed to a beginner, I'll try to use "easy" perl code that may easily be enhanced.

If I'm playing around in my sparetime and writing some codes propably nobody else is interested, I play around with perl, write programs that perhaps (sometimes hopefully :-) ) will never be finished, just to find out how I could do it. As soon as I've found a way, it very often isn't interesting any more.
With such perl-"games" I improved my perl-skills a lot, found a lot of perl "bugs" which very often turned out to be nice features, or found out about some nice techniques.

Best regards,
perl -le "s==*F=e=>y~\*martinF~stronat~=>s~[^\w]~~g=>chop,print"