Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

A Cautionary Rant

by footpad (Abbot)
on Apr 09, 2001 at 20:55 UTC ( #71067=perlmeditation: print w/replies, xml ) Need Help??

Subtitle: When Communication Fails

A year ago, I had a project dropped in my lap: write a CGI script that lets a user enter certain values and upload a file. After verifying the data and the file, email it to a certain address.

The project had been in discussion for 18 months, going through the prioritization, budgeting, overall design, and other administrative processes endemic to large corporations and their resource planning infrastructures. The original developer assigned to the task had, during that time, left the company. Due to other changes going on, so had the backup person.

This meant that there was one person left who had any interest or experience (heh) in writing CGI scripts. Me, who doesn't even belong to IT or IS. (I work for Marketing, which is another story itself.)

Thus, the project was dropped in my lap with a two week deadline. Naturally, I have zero access to the servers hosting the company's CGI scripts. Also, this is my first "real" Perl project.

I cobble something together within the deadline, test it on the server hosting my personal websites, and submit it to TPTB. Yay...successful completetion, right? Hah.

The project sponsor does some testing, asks for some enhancements, and starts trying to get it deployed on the company's servers. I quickly make the changes.

Interlude: I find the Monastery and begin to learn just how awful the original implementation is. I desperately want to rewrite it to make it better.

Three months later, Corporate says they're ready to test. And, oh, could I make a small change? Sure, no problem. Done, documented, zipped, and sent. I'm also able to slip in a couple of key changes. Lots still to do, though. But, I leave it alone as it at least works.

Four months after that, Corporate asks where the files are. I tell them I already submitted, but here's the ZIP file again, just in case something happened in transit.

In January, the new Product Manager asks me to attend a meeting on getting the project finished and deployed. (The original sponsor having left the company some six months before.) I attend, finally learn who all the players are, and am finally able to talk to the actual administrator who's just been hired to handle this stuff.

He asks, "What do I need to do to deploy this?" "Well," I say, "if you check out the README file, there's a pretty detailed list of configuration changes that you'll need to make, since I didn't know the names of the servers and no one wanted to tell me." (Okay, I was a little more diplomatic than that, but...that was was gist.) While we're discussing this, I hear confusion in his voice.

"Tell you what," I offer, "why don't you give me a week to see if I can't make this easier on you? This will give you some time to install some CPAN modules you'll need. There's a list in the README."

"Okay," he says. "That'll be fine." We tell the product manager and I finally have some time to make the improvements I've been wanting. Yay. I'm happy.

I clean things up, separate the configuration, add file locking, etc, etc. It's much better. I revise the documentation (ripping out some ten pages in the process), package things up, and send it off. I feel much happier thinking that the script is closer to what we like to see in such things within these walls.

Things languish in limbo for a time, in part because the Product Manager goes on extended vacation without telling the rest of team. I keep tweaking, just in case another chance to update appears. One does; I do so. It's still not perfect, but it's defensible. Once again, I revise the docs and sent it off.

A month later (today), I hear the Admin has finally gotten the modules installed and that we're finally ready for testing. ("Excuse me?" I want to scream, "Haven't you heard of -MCPAN?" I restrain myself...barely.)

During the same meeting (a conference call split across three cities), I observe that the:

  1. Product Manager has no clue who's responsible for what or what needs to be done to do it.

  2. Server Administrator has run into problems without consulting with Development (me). Furthermore, he blows off all meetings regarding the project.

  3. Security Administrator knows how to implement .htaccess and SSL, but has no idea how to deal with Perl or CGI scripts.

  4. HTML Designer is capable, but powerless because he has to beg Security for the URL's, server names, and other administrivia needed to connect a link to a CGI script.

  5. Development finally gets authorization for shell access to the deployment server.

The morals of the story are many, as are the mistakes (including many of my own). However, this is worse than any Keystone Kops flick or Fox "reality" show. Please notice the ones relevant to your work and take steps to avoid these types of problems.

Here's the main ones from my point of view:

  • Foster an environment of communication.

  • Clearly document (and make available) the names, phone numbers, email addresses, and responsibilities of the team members--especially if you're working with other teams that are not located in your offices.

  • Give people very clear guidelines of your expectations and needs.

  • Don't be afraid to ask for help, nor to kick some fannies to get things moving.

  • Give your people the training they need to do their jobs properly. At the very least, when approached by someone who wants to help with any problems, take them up on the offer.

  • Don't get so paranoid at avoiding social engineering that you exclude internal people trying to learn and work with your security practices.

Sorry...I just had to rant.


P.S. I got dinged on my performance review because this project is now a year late. No matter that I had my piece completed on time and under budget. G-r-r-r-r-r...!

Replies are listed 'Best First'.
(Ovid - communication rocks!) Re: A Cautionary Rant
by Ovid (Cardinal) on Apr 09, 2001 at 21:37 UTC

    Boy oh boy, do I feel for you. I've been there and, in my best Clinton voice, "I feel your pain." Typical example:

    In one company I worked for, we had a huge project that was supposed to automate the creation of what we called "small group plans." I was brand new with the company and, even though I only had a small piece to work on, I felt overwhelmed by everyone tossing around business terms that I didn't know. I talked to my boss about this and he advised me on a course of action.

    I spent a couple of weeks arranging personal meetings with various people who had free time and asked them to explain what they did for the company and how that might affect what I was trying to do (my job was to extract billing data from a agent/broker commission system and format it so that it could be read into an Oracle database). Once I had everything I needed, I wrote up the requirements and got everyone involved to sign off on them. I wound up with a very clear understanding of what was needed and wound up finishing my part about a month early.

    Several times after that, the project manager would call me up and yell that my program was spitting out garbage. Research into the problem invariably indicated that systems that fed into mine created the garbage, but that my work was fine. By the time the project was winding down, we were 6 months overdue and (according to rumor) about a million dollars over budget. Nothing in the system seemed to work properly. In fact, there was only one piece that I knew of that never needed to be changed: the susbsystem that I wrote. It worked because I didn't know what I was doing and took the time to ask everybody and their dog what was going on.

    The end result of all of that mess? "Small group coverage" was to be canceled as unprofitable. Our entire project went down the drain and the project manager was let go.


    Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.

(dws)Re: A Cautionary Rant
by dws (Chancellor) on Apr 10, 2001 at 01:19 UTC
    You've identified a number of project-leader level problems, though it looks like the project was suffering from a bigger problem that you might not have has as much visibility on:

    No Effective Executive Sponsor

    The role of (or even the existence of) an executive sponsor is often invisible to the developers in the trenches. An executive sponsor is the person on the executive staff who is motivated to see a project through to completion. They cut through high-level roadblocks and keep things moving. Projects without sponsors tend to drift around, subject to corporate weather patterns, until some momentary crisis gives the project temporary momentum. Lots of people (including Security) will happily say NO to you unless you have a sponsor.

    Here are some warning signs that indicate that the project had a serious sponsorship problem:

    1. The project took 18 months until a development task lands.
    2. The development task lands not in Engineering, but in the hands of someone in Marketing who didn't yet speak Perl.
    3. Seven months later, after doing some tests, Corporate has lost the handoff.
    4. In the meantime the original sponsor has left.
    5. People think they're powerless, and unable to get vital information from Security.

    With the possible exception of Corporate losing the handoff, these aren't project manager or product manager level problems. If they were, an effective sponsor would have had the the project/product manager replaced.

    I try never to get involved with a project that doesn't have a real, live, politically protected executive sponsor. Life is too short to be a ping-pong ball.

      You mean everyplace isn't run like this? I guess I drew the short straw then...

      perl -pe "s/\b;([st])/'\1/mg"

Re: A Cautionary Rant
by voyager (Friar) on Apr 10, 2001 at 05:50 UTC
    Warning: if you are young and idealistic, read no further.

    The situation described is pretty common. Learn to deal with it or go off and build something for yourself.

    I was in a talk by Phillip Greenspun last week. One of his many quotes was, "Human beings tend to be imprecise, but computers demand precision". People who gravitate to computers appreciate precision. Those who ask programmers to do things, don't appreciate precision. And can't be rewired.

    My current boss has described it as: "Most people have no imagination and can't read (specs). They will only understand what you are going to build when you show them the finished product, or a reasonable protottype."

    You should meet, you should document, you should foster a team atmosphere. But in 20+ years of programming and many companies (oil, banking, research, dotcom, web design, etc), it's still the same: Programmers sitting around complaining about the people they try and please.

    Bottom line: most of the time your customers won't appreciate what it takes to get the job done. They may be happy with the result, but they won't appreciate what it took. If you don't want to go crazy, you must accept this at some level. And come here and commiserate :)

      While there is much truth in this, I've found that too many programmers refuse to learn even the fundamentals of the subject matter of the business. Many will tell you this up front and without shame. I know many programmers who have worked at my employer for years and who still do not know the first thing about the business.

      In my job, I write requirements for others to code, and I write code from others' requirements. (Often, the entire development cycle begins and ends with me.) In the former role, I often find myself writing requirements so detailed that they begin to resemble pseudo-code because I know that the programmer lacks the business context that would facilitate understanding. Sometimes I start to feel as though I should finish the design and code it myself. If I have to spell out my requirements to the degree of precision that a computer understands, why should I hire a programmer?

      My experience is that for every business type who's unwilling to understand the need for precision in requirements, there's a programmer who's unwilling to learn enough about the business to understand its needs. And I, like many other business types in the finance industry where I work, probably appreciate the need for precision as much as any programmer.

      While I do agree with the essence of this post--that communication is critical--I think it's important to remember that there's more than one side to this story.

        michael you are correct. There is as big a problem on the development side, but I was responding to the poster who appeared to have his act together.

        In fact I hold developers more responsible for not holding up their end of the bargain because we are supposed to be more logical thinkers, etc.

        I read a recent article that questioned the actual success of eXtreme Programming (XP), but one of its fundamental concepts is that end-users are a core part of the team, constantly interacting with the developers and the application as it is being built.

        UML (nor any other documentation/specification scheme) simply can not describe an unbuilt application to the degree that a blueprint can describe an unbuilt building. I believe the biggest barrier to successful software is users unable/unwilling to spend sufficient time with developers through the life of the project (this is of course moot if the developer can't/won't interact with the end-user frequently and in jargon-free language).

Re: A Cautionary Rant
by Fastolfe (Vicar) on Apr 09, 2001 at 21:42 UTC
    Sounds like bad organization. You need a Project Manager. Someone that gathers business requirements and coordinates between developers, administrators, security and the client. All changes and enhancements go through the project manager. The developers are approached from and communicate with, chiefly, the Project Manager. Naturally there will be details that require some direct contact between the various parties, but things like change requests, deadlines and all of the coordination would be handled by the PM. Regular meetings are good too, and those that chronically fail to attend should be disciplined if the project suffers as a result.
      I personally work for a small startup ( VERY SMALL ), and even in small companies this type of thing happens.

      A perfect example is the project we are currently working on. Even with 2 programmers, there are a number of steps that we have to go through to code up and running.

      Get the project description.

      Write some code

      Have the results rviewed for changes ( and for some reason, 90% of the time major "oh, can you add this" additions occur )

      Write more code, and rewrite/change 50% of code

      Get reviewed again

      write more code... and rewrite more code

      Get reviewed again

      HTML guy takes over ( all code we produce is templated , other then table internals ect )

      Make a ton of changes for HTML guy

      Reviewed again, now with the HTML

      write more code...

      It goes on and on...

      It seems to me , no matter how big or small the company, an inordinate amount of time is spent developing the project compared to actually developing the code.


        That can be very true, the problem is I think.. most times ppl. don't understand or thoroughly understand the whole project until at least half the prottype is ready to be shown to the upper management to gain approval/critisims/suggestions. Sometimes developing the project can be half the battle of a project.. Fastolfe is right about a Project Manager - it does help. It leaves the project leads and other developers free to do their job and allow the administrative tasks to be handled by someone else who is outside of the project development(coding that is..), but who understands what the project must do.. they sometimes can also coordinate with the clients to gain feedback or be the main person of contact via the developers. But, knowing the project and developing it to specs(no matter how many versions down the road) takes quite a bit of time - and communication is really key.
Re: A Cautionary Rant
by stephen (Priest) on Apr 10, 2001 at 07:04 UTC
    You might be interested in the book Anti-Patterns: Refactoring Software, Architectures, and Projects in Crisis. It's an excellent guide to the sort of debacles you describe.

    (In what follows, I'm describing mostly what I've sworn to do after a bunch of experiences like yours. Customer results may vary.)

    The only advice I have to offer on that kind of thing sounds cynical, but it's the best I can do: COVER YOURSELF. If someone fails to deliver something for you, send them an e-mail-- and CC your boss. Keep copious notes of every meeting, and if anything is agreed on in the meeting, send e-mails to all persons affected stating what was agreed upon, even if they were sitting right next to you when the agreement was made. Documentation can be your friend in performance reviews.

    Also, it's best to figure out-- and keep notes on, in a safe place-- what each participant has to gain and lose by the project getting done on time. For example, the Server Administrator's career is not going to be helped if your project is a raging success, nor will it be hindered if it fails. People who aren't directly affected by the outcome of the project generally can't be depended upon to help it. The only thing to do is to MAKE their lives affected by the outcome.

    Anyway, it was a very valuable rant. Wish I could ++ it twice.


Re: A Cautionary Rant
by delegatrix (Scribe) on Apr 10, 2001 at 05:00 UTC
    Ouch. SOunds like your organization can use a good web server staging machine, too, to duplicate your production system. Having to ask security for URLS? Sheesh.
Re: A Cautionary Rant
by fmogavero (Monk) on Apr 10, 2001 at 23:22 UTC
    You are not alone.

    Hi, my name is fmogavero, and I'm a Perl developer.

    I have learned to render unto Caeser what is Caesars. My projects are buried in mounds of paperwork.
    Even though my DBConnection object (please don't ask why)is only 81 lines of code, the accompanying paperwork is 81,000 lines of garbage, dated, categorized and "submitted for your approval. (Even though we all know it is much easier to beg for forgiveness than ask for permission)

    I read the list of morals and tears of joyous sympathy flowed freely down my cheeks. When are they (business) going to learn that if they want information technology that rocks, they need to listen as well as demand. Where would they be right now if all the computers went on strike? They need to learn about the power outages in California. (And they need to learn how to use a Soroban or Abacus)

    You are not alone. I've been there, done that and got the t-shirt too. I am just sorry that it happens to others too.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlmeditation [id://71067]
Approved by root
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (5)
As of 2021-10-16 17:57 GMT
Find Nodes?
    Voting Booth?
    My first memorable Perl project was:

    Results (69 votes). Check out past polls.