|Problems? Is your data what you think it is?|
Re: (OT) Programming as a craftby BrowserUk (Pope)
|on Dec 16, 2003 at 04:18 UTC||Need Help??|
I drew my conclusions on this subject when it came up once before, and nothing has changed in the meantime except that I ended that piece with
Hopefully, that will remain the case at least until after I retire.
A change of mind, or maybe heart
On this I have changed my mind. Over the intervening year, I have had the time and motivation to look at where I think we are on the evolution of both the business and the profession of software development, and I have reached the conclusion that not only will it's current 'craft' status change -- it must change.
Despite my natural reluctance to see the end of what one might emotionally call 'a way of life', the one I have pursued with interest and fun for 20+ years. I now see it's demise as not just inevitable, but desirable. Whilst I previously thought that I might get away with being a Luddite and basking in the glory of the 'lone craftsman' persona until I retire. Thanks to having had the time and motivation to take a look at specific aspects of what we do every day as programmers and look at them in detail -- something that as a working programmer I had never had the time to do before. Not just 'performing the functions of a programmer', but looking closely at how the methods we currently use, evolved, and where they might lead to if pursued (with some hopefully intelligent guestimation and projection) into the future a ways.
A tentative conclusion
I've reached (a tentative) conclusion that we, both collectively and individually, are on the cusp of a major leap in the way software development is both performed and it's status as a result. The change I foresee, might be called, for want of a better term, automation.
Just as with so many other mass-market products that start out being manufactured by small groups, usually driven by highly motivated, and not necessarily "professional" individuals, the transitioned to mass manufacture by a small number of very large corporations in order to achieve the benefits of scale required to fund the huge R&D costs for 'new models' is inevitable. That will probably not be a popular conclusion. The chances are that those with a little knowledge of history will envisage software moving into the 'production line' phase, with the inherent horrors of individuals performing endlessly repetitive, menial tasks in lock step. With all the skill, flair and therefore interest diluted or completely removed. Like an auto-manufactures assembly line. I too used to think that this was the next step on the evolution.
After all, this was the case in so many other industries. And not just the obvious examples either -- the auto industry, motorcycles, TVs and other electronic consumables in general, even the PC. With a little sideways glancing at many other industries, you can also follow this progression. Tourism & air travel, with package holidays, mass-market cruise ships following each other on the same schedules just far enough apart that the wake from the preceding ship doesn't 'shake the contents'. A little harder to see are things like music, film, fashion and even art. However, I think that there is a major difference that means that all is not lost. The older of the above industries have moved beyond the assembly line stage, with thousands of people arranged either side of a moving conveyor performing repetitive tasks like automatons. They have been replaced, by automatons -- robots. The driving force behind that replacement was costs, mostly labour costs. The enabling technology that allowed the replacement was computers. These gave the machines that actually performed the task of assembly, the intelligence and dexterity to perform those tasks without the guidance and supervision of the human brain. This is the same technology that will (IMO) allow the software development industry to bypass the assembly-line-powered-by-humans stage and move directly to the automated assembly line stage.
Essentially, the craft-to-assembly-line transition involves breaking the task of overall assembly down into infinitesimally small, discrete steps and then assign each step to an individual (human), to master and repeat endlessly. The enabling technology for this in the mechanical engineering field was the use of the basic had tools of that craft -- scribes, rules, micrometers, hammers, saws and files -- to be used to create better machines -- lathes, routers, presses etc. Once this was mastered, then it became possible to create specialist versions of these machines that would produce sub-assemblies that the individual workers then assemble together to form the final product. This is roughly where we are today in the software industry. We have the second level of tools at our disposal -- editors, compiler, interpreters, version control, CD burners etc. With these, we can produce the software sub-assemblies, classes, libraries, objects etc. The industry is now trying to get to grips with the mechanisms of utilising these sub-assemblies in production line environments -- the moves towards offshore software development are the current signs of this although there were plenty of corporate MIS depts. that attempted to do this kind of development throughout the 80's and 90's. Assuming that throwing large numbers of programmers at tasks, under strictly controlled development team regimes would result in quicker development times and reduced costs and reliable time scales. Personally, I think that both of these strategies will eventually be discredited and discarded as failures. And not before time. The next step is robots.
A ray of hope
I can sense (or pre-sense) the hostility that many who read that statement with feel. They, like me, do not want to give up their job, their skills, their art, their passion to a robot. Fear not, you won't have to. The reason is that, as yet, we have not for the most part reached the point where our jobs can be automated, not even by computers. More importantly, we have not yet reached the point where the numbers of people employed in the industry and their total salaries have become such a high proportion of the total available income from the potential market for our products, that they govern the profitability of the industry as a whole. With cars, the markets where pretty much saturated in terms of numbers somewhere in the late 80's, early 90's. As such, the only ways to get higher profits was to
The software industry is far from reaching this point. In fact, the limiting factor in the applications of our products is our ability to produce them. There are a million new uses that software could be put to, and the new markets and uses grow every day, as the hardware gets cheaper and more powerful.
The limitation that stops the field growing is the productivity of the industry. It simply takes too long, using current techniques, to bring new products, or even better versions of existing ones to fruition. During the 80s and again in the dot Com bomb of the late nineties, as the market place for our endeavours grew, the attempt was made to ramp up production by increasing productivity through assembly line programing and by recruiting large numbers of low-skilled personnel into the industry. Both failed.
The lesson that can be learned from history of other industries is that the only, long term, effective way of increasing the productivity of an industry, is not to dumb down the job and throw bodies at the problem, but to increase the skills of the existing levels and have them develop tools that perform the mundane and repetitive parts of the process. A small number of highly skilled personnel use their skills and innovation to automate those parts of their jobs that they hate. Universally, the boring and repetitive parts. The number of people employed remains static, but the education (and salary:) levels increases as they use automation to multiply their productivity. I recently typed the following in another post here
The secret of productivity is to have tools that do more of the work, more quickly, more accurately and with less supervision. Computers, as tools, are being applied in every area of human endeavour to this end with amazing results. With one, notable and lamentable exception -- software development.
I repeat it, because I don't think I can say it any better. I think that this is not only the indicator of what we in software need to do, I also think that the time is nigh when the technology, the motivation and even the financial climate are such that we will be able to do what it necessary. And that is to produce better tools. That doesn't mean better editors, or faster sort algorithms, or easier to use versioning systems, or more testing, or bigger, more comprehensive code libraries. It involves the integration of all of these and more. We need tools that automate steps, and facilitate that automation. We don't just need a bigger parts catalog. We need tools that know about the parts catalog and help use to select the right parts from it. We need to be able to select not individual parts and glue them together, but to be able to specify the purpose of our applications at a high level and have our tools select the appropriate parts for use and integrate them together for us. We need to be able to interchange parts on the basis of performance as measured by any of several criteria and pick the one that supports the particular performance characteristics required by our application.
We specify a sort. If the application requires a faster one, we swap in a faster one, or a stable one, or one capable of handling large volumes of data, or one that is specialised in dealing with the particular attributes of our data -- transparently to the rest of the application. Perhaps the application can select the sort it uses on the basis of real time intelligence of the data it has to sort. The tools, and even the applications themselves have to become more intelligent in selecting the building blocks they use to perform the tasks at hand, given real-time intel on the nature of the data upon which they must perform it.
Ultimately, this probably requires more intelligence in the software that runs the tools and the applications. Ie. The OSs themselves. We need to move beyond the current crop of models of the world and the data in it, which evolved in the 1970s and 80s to model the world in terms of the architectures and hardware, and their limitations, available at that point in time. Higher level languages is one part of this. The integration of those languages with data stores that don't reduce the world to a series of bytes streams is another.
Exciting times (I hope)
In many ways, I see this as the most exciting time in software development. Watching the tools and systems that will integrate the innovations of the human mind with the power of the hardware now available to us, will be a privilege to witness. If I can find a way to take part in the process, so much the better.