Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

A Porting Horror Story

by stephen (Priest)
on Apr 09, 2002 at 22:10 UTC ( #157876=perlmeditation: print w/ replies, xml ) Need Help??

Once upon a time there was a small company that had a great deal of legacy code written in Perl. The new engineering manager and the new CTO wanted to move to a Java-based solution. They had several reasons for doing this:

  • Neither person knew Perl.
  • They wanted to impress their investors with the "Java" buzzword.
  • They felt the code was so horribly-written that it needed a ground-up rewrite before they could add features.
  • They believed that Java was more "scalable" than Perl.

They hired several Java programmers (none of whom knew Perl) and began to rewrite the code from the original specs. They estimated that it would take them four months. However, the specs were not detailed enough, and the resulting product was rejected by quality assurance, since it would not meet existing customer needs. All told, it took a year to complete the rewrite, during which time features needed to be added to the legacy code to support customer needs.

This experience changed my opinion on porting and rewriting. At the onset of working on this project, I believed that after a certain point code needs to be rewritten from scratch. I believed that if a design is flawed, there's no way to patch it to make it work.

My new rule is "first refactor, then port." I base this rule on the fact that no specification is detailed enough to specify all of the operations of a piece of software-- and if it is, it'll be outdated once the program enters maintenance mode. To port software from one language to another, it's necessary to make the code comprehensible in the existing language through refactoring. If the design of the original software is bad, you can use refactoring to fix the original design. Once the refactoring is completed, you can (and should) reevaluate the necessity of porting.

Let me go over a few of my problems with the reasons for making the decision to port in the first place:

Neither the CTO nor the engineering manager knew Perl.

This disturbs me a great deal in retrospect. Certainly implementation is not the job of the CTO. Nor should it necessarily be the job of the engineering manager. Even if a company were so resource-strapped as to be unable to hire Perl programmers, it takes far less time to learn a new language than it takes to rewrite a complex program from scratch in a different language without reference to the original code.

They wanted to impress their investors with the "Java" buzzword.

I think Time has proven that investors are more impressed with the "profit" keyword.

They felt the code was so horribly-written that it needed a ground-up rewrite before they could add features.

This is the objection with the most validity, but even it rings hollow. Refactoring techniques have become more and more widely accepted. Plus, refactoring has the virtue of keeping the code running while one improves it and adds a battery of automated self-tests. Refactoring the existing solution concentrates the effort onto the current code, rather than splitting it between maintaining the current code and writing the new code. This lack of concentration was a major problem at our company, where a skeleton crew had to maintain the existing code while the bulk of Engineering staff worked on the new Java code.

They believed that Java was more 'scalable' than Perl.

One hears the 'scalable' buzzword more and more often these days (well, these days I more frequently hear 'downsize' and 'pay cut', but hey). 'Scalability' is one of Java's perceived attributes. However, when one tries to get a meaning for 'scalability', one learns that it means many different things to many people. My original understanding of 'scalability' was that scalable software did not suffer from hardware bottlenecks-- that as demand increased, that one could purchase additional hardware to handle the stress, and that the software could benefit from this. Such a characteristic is a design feature, though, not a language feature-- one could write such software equally well in Java and Perl.

Another meaning that I hear for 'scalable' is simple efficiency-- that Java runs more quickly than Perl on the same hardware. However, even if that's true, what sort of logic leads to porting to Java for efficiency reasons? If you're going to port for efficiency, port to C. Better yet, refactor the Perl code, then take the most frequently executed module in that code and port *it* to C, then link it back into the Perl code as an XS module. That way you get a much faster gain for far less effort. (Porting a Perl module to C is definitely not a trivial undertaking, and may not gain much efficiency. But a possible failure here is much less risky than the nearly guaranteed failure of a large-scale porting project.)

Conclusion

The small company that I describe eventually managed to complete their porting project. It finished more than a year behind schedule. They had worked frantically for more than a year just trying to get the Java code to operate to the Perl code's spec-- while at the same time customer demand was forcing changes in the running Perl code. At the same time, the speed at which they were trying to port this code forced them to cut corners. Their Java code wound up just as difficult to understand as the original Perl code.

The lesson I have taken from this escapade is that it's rarely a good idea to do something twice-- even in a different language. I believe that the lessons here would apply equally well if the arrow pointed the other way, from Java to Perl. It even applies for many ground-up rewrite projects that remain in the same language.

Finally, there are many good reasons for porting. If your software simply cannot run on its intended platform, you don't have a choice. However, even in that case, you will do well to refactor your existing software as much as possible before porting.

stephen

Comment on A Porting Horror Story
Re: A Porting Horror Story
by dws (Chancellor) on Apr 09, 2002 at 22:23 UTC
    In addition to the sins stephen calls out, the engineering manager and CTO committed another major blunder: They assumed that the written specs reflected the existing Perl codebase. Bad move.

    Faced with a legacy codebase and legacy specifications, trust the codebase.1

    Specs lag implementation, unless one puts a lot of effort into keeping them current -- effort that code be applied elsewhere. By ignoring the existing code base as a source of truth, they probably dribbled requirements all over the floor.

    ____
    1. This is a variant on another useful saying: "When the map and the territory don't agree, trust the territory."

Re: A Porting Horror Story
by japh (Friar) on Apr 09, 2002 at 22:24 UTC
    I blinked at the use of 'legacy' and 'perl' in the same phrase, as I'm used to thinking of perl as a relatively new language, but a quick mental inventory of our network made me realize that we in fact run legacy code as well -- but only because it is still doing what it was originally designed to do, not because it's unmaintainable. The bits that have needed to change over the years, have.

    A question that came to mind while reading this post: does anyone have a grasp on how many active perl programmers are out there, versus the number of active java programmers? (I mean folks who code on a regular basis and have progressed farther than 'Learn to program X in 21 days' level.) A factor in deciding which language management should put their energies behind would be the size of the pool of programmers available for the task, from the standpoint of both cost (higher supply generally lowers the price) and community resources.

      See, japh, we all know Perl is more popular, but let's just keep that info to ourselves, shall we? I really don't feel like another pay cut. ;-)

      -Mike-
Re: A Porting Horror Story
by Necos (Friar) on Apr 09, 2002 at 23:29 UTC
    Whenever I hear these kinds of stories, I think to myself: "I'm glad I don't have to deal with that madness." When I code utilities in Perl, I code them for the task at hand, leaving room for improvement (read: expansion). This is, from what I understand, typical software engineering. If not, it is definitely a good habit.

    Here, I grin at the fact that I work for a high school, instead of some PHB-type company. We don't have money to hire 9 million Java programmers, so we code our utilities in what can be done the fastest: Perl. That's why people like myself advocate the use of Perl. It's easy to learn (read: low learning curve) and get something very simple started. It's possible to extend code to be OS-specific if the need should arise. Also, it's one of the few languages (that I know of) that allows for the use of modules. This saves me a lot of time. I can just import a module into my program and start writing.

    I don't advocate the use of Perl because I'm some hardnosed Perl hacker. I advocate the use of Perl because it gets the job done, plain and simple. It may not be the fastest thing since C, but that's fine. If I was really worried about speed at that level, my machines would not be running Windows.

    In short... Use the language that does what you want when you want and makes it easy for you. Perl is my sword, and I try to use it properly. Until I reach the level of Perl hacker, I'll still train to better my skills.

    Theodore Charles III
    Network Administrator
    Los Angeles Senior High
    4650 W. Olympic Blvd.
    Los Angeles, CA 90019
    323-937-3210 ext. 224
    email->secon_kun@hotmail.com
Re: A Porting Horror Story
by moodster (Hermit) on Apr 10, 2002 at 08:24 UTC
    Rewriting the whole code base from scratch is often very tempting when you're knee-deep in APIs that started out reasonable, got changed due to changing specs, got changed again because one programmer felt like it, then got changed again because another programmer realized that the API didn't actually work. And so on.

    You say to yourself "Bah, I could rewrite this from scratch in a week and still have time to leave early on friday". You convince your boss and your colleagues that you can do it and they give you thumbs up. You sit down monday morning, pour a cup of coffee and start to type away. And then you're screwed.

    The thing is, a lot of thought has gone into you existing code. It looks ugly, yes, but that's because it already contains the bug fixes, the workaraounds and the quick hacks that make the system run in your choosen production environment with all its limitations, known issues and bugs. If you start over, chances are that you're gonna have to reconstruct all this accumulated knowledge all over again.

    That said, and with the java-bashing going on in this thread, I'd like to publicly state that I love java. Thank you.

    --Moodster

      moodster++

      I also love Java, and didn't mean to make an attack on the language. Java has its good points and bad points, like any other language.

      In their defense, the folks at this company were all pretty bright people. They made some bad decisions, but the decisions looked good at the time. Hindsight is easy.

      Also, the decision to refactor is a really tough sell. Imagine going to upper management and saying, "Hey, I've got a great idea-- the Engineering department is going to spend about half of its time making changes to the code which have no effect whatsoever visible to the outside world. But after we do that, the code will be mysteriously better!" Porting, although it is really the same thing in a much bigger chunk, gives upper management something they can point to. "See? We're moving forward! We've moved this from Language A to Language B!"

      The disaster described here could have happened just as easily porting from Java to Perl as from Perl to Java... or even doing a rewrite-from-scratch on a project in any language.

      stephen

Re: A Porting Horror Story
by stefan k (Curate) on Apr 10, 2002 at 10:55 UTC
    Hi,

    They wanted to impress their investors with the "Java" buzzword.

    Without wanting to start a flame war or offend anyone, just because I love the cite: Java is a hype-oriented language.

    More serious:

    • What's so bad about legacy code when it does what it was written for? Here at my company one of the oldest running programms (I'm pretty sure it is _the_ oldest ;-) is a very, very ugly, totally unmaintainable perl script I wrote when I just had started to learn perl. Well, it does it's job for years now. OK, if a feature request would arise I'd probably rewrite it from scratch, or I'd try to use the existing programm more as a module and try to wrap something around it.
    • We can see another point here: we programmers still have the habit of overestimating our skills; read: understimating the problem at hand and the time it'll take. It is no wonder that so many project die away and very often the coders later blame the management (or someone else in a similar position) for the timelines. Well, quite often those timelines are based upon a coder telling his team-leader that it is not a problem to write this or that in a few days. I have seen projects which had dates and milestones which could not be reached and which were proposed by "leaders", and I have seen coders speaking of projects that will have a few thousand lines to be done in a week. Well, Yeah ... Come on...
    • And finally: your story contains the old legend of correct specs which were laid out before a single line of code was written and which precisely describe what the programm is supposed to do in the end. *laugh*. I must admit: I have never seen that in real life.

    Regards... Stefan
    you begin bashing the string with a +42 regexp of confusion

Re: A Porting Horror Story
by meonkeys (Chaplain) on Apr 10, 2002 at 19:53 UTC
    Great post. Something like this probably qualifies as an axiom: "understand why you want to rewrite something, and understand what you may be rewriting".

    Another issue that I haven't seen touched on yet is the new CTO's ego and political motivations. I would guess that if a new CTO just came in and shot par for the course by keeping the legacy system running well, that CTO wouldn't have much to add to their resumé. It seems like it makes more slimy political sense to say "Everything sucked until I came on board and made it better"; especially if it's not true.

    ---
    "A Jedi uses the Force for knowledge and defense, never for attack."
Re: A Porting Horror Story
by jlongino (Parson) on Apr 10, 2002 at 20:13 UTC
    I think the most disturbing/humorous aspect of this story derives from the following:
  • Neither the CTO nor the engineering manager knew Perl.
  • They felt the code was so horribly-written that it needed a ground-up rewrite before they could add features.
  • They believed that Java was more 'scalable' than Perl.
  • If the first is true, how the heck did they arrive at the other two conclusions? Sounds like PHB voodoo to me. Perhaps they should've hired merlyn for a code review. :)

    --Jim

Re: A Porting Horror Story
by Molt (Chaplain) on Apr 11, 2002 at 16:09 UTC

    Thanks for posting this, even though it made me shudder. I'm part of a team approaching the end of rewriting a major part of the infrastrucure of our search engine, and it's gone remarkably well but reading this I see how we could have fallen down dramatically.

    The majority of the team here are Java programmers, and our front-end is all J2EE stuff. I have my own views on this, but the fact is it's here and it's not going to change any time soon. The starting code for the section we were rewriting was out-of-house Perl though, horrendously mangled in my opinion and desperately in need of either a rewrite, or at least a major tidy.

    To start with we knew what we didn't want to do, we didn't want to just clone the system with it's annoying quirks and all. We wanted to tie parts more closely in with the front-end business objects to avoid having the same code concepts duplicated in two places in two different languages, with the attendant maintenance nightmare.

    The result is remarkably clean considering the amount of detail and 'special cases' we had to deal with. The initial pre-parsing is all Perl, which then hands over to Java to do it's bit of the work, and finally back to Perl to merge the Java programs results and replicate them to all of the servers.

    The difference here was we had a team leader who knew what he was doing, we had a team who despite having very different core competencies managed to hold together, we had people who knew what the old system should do despite being unable to read the language it was in, and me who knew the language but didn't know 'what it did'. We even ended up leaving small parts of the old code on the basis no one knew them well enough, and if they collapsed so did the search engine.

    We went out there with a view to make this work, in-house, and don't get hung up on language terrotorialism and buzzwords.

    Thank you, Stephen. It's nice to see how lucky I am at times, just hope you have the luck to soon get a job where you'll have your own success stories once more.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://157876]
Approved by Ovid
Front-paged by cjf
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (5)
As of 2014-07-29 01:03 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (211 votes), past polls