Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Specify, Specify, Specify

by Ovid (Cardinal)
on Oct 01, 2001 at 05:01 UTC ( [id://115778] : perlmeditation . print w/replies, xml ) Need Help??

Most of us hate to write specifications. They're seen as an onerous task and, for many small projects, an irrelevant one. Whether or not the task is onerous is purely a subjective opinion. Whether or not the specifications are irrelevant is not subjective.

Case in point: a simple form-to-email script.

My company is offering a product that has generated a huge amount of interest. In order to handle the unexpectedly large demand for information, we wanted to include an information request form on the Web site. I happened to hear my name mentioned by my boss and wandered over to catch the tag end of a conversation that was about what I needed to do. Coding a script that captures about 20 form fields is not particularly difficult. Untainting all of them can take a bit longer. My boss knows that I use a form-parser that I've written to automatically generate the form handling code. Thus, he assumed that it would take me about 15 minutes to write the script.


It actually took me about three hours, mostly because no specifications had been written. I was told to simply write a form-to-email script that would send us an email with a request for 8 by 10 color glossy photos (thanks, Arlo!) of our product. I was sent the link to the HTML form and that was my complete specification. Nothing, of course, was in writing.

Now many of you are probably thinking, what's Ovid blathering about? That's a piece of cake. In reality, it's a bit more work than it appears on the surface. Further, because no specifications were given and because the people who dreamed up this little assignment took off for a couple of hours afterwards, I was left to come up with the specification on my own.

I decided that the form had to be "sticky" (information filled in if the user is sent back to the form) because the user may not supply all of the necessary information. I went to the person who designed the form and requested a confirmation page (one would think they'd get the hint, since I request this for all pages where there's no way to verify that an action had really occured). Further, since I try to be sensitive to privacy information, I made all contact information, except for the contact name and address, optional. We needed to mail a brochure, but if someone doesn't want an email or follow-up phone call, then they can "opt out". I also realized that the server this was to be on didn't have Template Toolkit installed, but did have HTML::Template. However, no one here uses HTML::Template and I didn't have the authority to just go out and install the Toolkit. What do I do? Use "HERE" docs? Ugh!

On my first iteration of the script, I reluctantly used HERE docs because I had no idea what was intended. I got the confirmation page from the designer and had everthing running fine. Sticky fields, useful error messages about required fields, and a slick confirmation page were thrown together pretty quickly. Then my boss got back.

As it turns out, there were three completely different forms that I needed to capture information from. My required fields were wrong as we needed phone numbers and email to bug the heck out our potential customers with. And I could forget about the HERE documents as they want Template Toolkit on that server, anyway. I had a modest amount of my code that was now junk and had to be rewritten. A simple form-to-email script that should not have taken very long wound up taking several hours to write because I was forced to guess what was needed. Had specifications been written, I wouldn't have wasted my time and I could have gotten back to my regular work much sooner. The next time you're thinking about skipping clear specifications, think about this. If it's more complicated then the script that I described, how many hours are you going to waste?


Vote for paco!

Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.

Replies are listed 'Best First'.
Re: Specify, Specify, Specify
by dthacker (Deacon) on Oct 01, 2001 at 07:17 UTC
    I grew up in programming in very small shops, where formal requirements were seldom, if ever used. I prided myself on my "customer service" and flexibility. If the customer wasn't happy with my guess, then I'd be happy to fix it. And fix it again ...As time went by the constant re-work that this mindset doomed me to began to wear on me. Projects didn't get done well, or even done. I began to see the users as adversaries, something I happen to believe is the kiss of death in this field.

    A few years ago I started working with an App Development Manager who insisted that end users come up with clear specifications before work began. Many of the end users were used to the previous style and would say things like "Oh, you just throw something together for me and I'll change what I don't like" This manager would politely point out that our (the IT departments) time was a scarce resource and was not going to be used in guessing games. And that, to me was a new way of thinking.

    Our time, knowledge, and talents are a scarce resource, whether we are more admin/DBA monks like me, or design wizards like Ovid. We need to use this resource wisely so we can be people who are smart, and get things done. Requirements cut down on re-work by exposing flaws in the business rules, or shortages of resources needed, before we attempt the first line of code.

    I've had end users react in various ways to the requirement of clear specifications. Some think I'm shirking my responsibilities and are making them do my job. Some become treasured allies in the design/build process. Some get lost in the details. Some never come back with a spec!

    In the end I can say we have less re-work, less finger-pointing, and more productivity. We also have much clearer communications between the users and IT staff. It's been a win for all.

      I still am in the small shops and we have taken advantage of the fast turn-around inherant in Perl to produce 0.1 style versions of what is requested. This prompts the user to respond with more details. A rapid developement cycle gets to the end result faster with more end-user satisfaction (especially if they feel they've been in on the consultation process). It's also a defensive manouver in an environment where we get a lot of spurious requests.

      The politics of the situation is that we have to be seen to deal with poorly thought-out demands, with few staff, or be labeled obstructionists to management by those with an axe to grind. Our best weapon is a quick response, placing the onus back on the user to specify what they really want. That, and a web-based requisition system that users are encouraged to use to track progress on their whim^Wrequests.

      Ea :wq

(Daddio) Re: Specify, Specify, Specify
by Daddio (Chaplain) on Oct 01, 2001 at 05:56 UTC

    Ovid, I agree with you completely!

    My 'regular' job just (and I mean just this past week) decided to add Perl as an accepted language, so all my 'projects' thus far have been without actual specs.

    But my larger problem is all the side work I do with a friend who runs his own ISP/hosting service. I can't remember one project on which I have worked that actually had complete specs. One project had something defined at the start, but then came the rolling-requirements, and just as I was ready to deliver phase one, things changed. And they continued to change. And grow. That wasn't fun. Now I get things like "I need you to look at some logs and tell me the security problems," or some such thing, with a snippet of a log file and nothing more to go on. That's a big help.

    I don't know where I am going with this except to say that I absolutely agree with you, and have already stated that any future projects need firm requirements and specs before I will start them. I have wasted too many hours and lines of code, and with a growing family I just don't have all that time to waste any longer. It hasn't helped so far (the logs thing was two weeks ago), but I am generally an optimist.

    D a d d i o

Re: Specify, Specify, Specify
by clemburg (Curate) on Oct 01, 2001 at 17:07 UTC

    You probably will want to read Software Fundamentals. Collected Papers by David L. Parnas.. In the papers in this book, Parnas not only gives a formal definition of what a specification is (and how a program can be said to fulfill it), but also contrasts it with other related concepts (model, prototype, description) and gives some useful techniques to improve the writing of specifications.

    If one goes straight to the commonsense meaning of "specification", a specification is nothing more than a statement of what we want to build. Given this definition, I have not seen any manager or customer resist working up at least a short specification for the work to be done.

    The real problem IMHO is to choose the detail level necessary for a specification. If chosen too deep, the specification will strongly resemble the program to be built. If chosen to high, the specification will be meaningless to the programmer.

    Usually, managers or customers will start with an extremely high-level specification. The trick seems to be how to get them to provide more detail, without swamping them with technicalities that they don't want to see and that they probably can't absorb.

    In my experience, focusing on these items will provide relatively useful specifications that can be agreed upon in minimal time (e.g., 15 minutes to an hour for a 2 day programming project):

    • critical success factors - what objectives the project has to fulfill to be considered a success
    • user screens (pencil drawing) or other example inputs and outputs
    • rough description of data to be handled from user point of view
    • any functionality that is *not* just create/update/delete for the data items (security is always worth a question)
    • performance constraints (interaction time for user screens, maximum batch loads, etc.)
    • budget available (time or money) - very important - this is your negotiation parameter!
    • who will make the decisions on this project (your direct customer)
    • contacts for further questions

    If a manager or customer resists discussion of these items, I usually try to get to his reasons for doing so (feelings of over-complication of trivial tasks, no time, not responsible, etc.) and to counter each one with a proposal on how to go on (explain I don't understand as much about his business as he does, ask to be given a date for a short meeting, asking who else can I ask, etc.). If really nobody can or wants to give you a specification, another option is to declare the project a "Research and Development" project, and to proceed in tight feedback loops with users and other participants to the project. Most of the time, a suggestion like this makes the argument for writing up a specification. The rest of the time - well, you have hit upon a nice playground - or you are being chosen as The Scapegoat (tm). Make sure you got a budget to play with to make the decision.

    Christian Lemburg
    Brainbench MVP for Perl

Re: Specify, Specify, Specify
by greywolf (Priest) on Oct 01, 2001 at 09:45 UTC
    I agree and I think we have all had to deal with this in the past. Most likely we will have to deal with it again in the future.

    The key is in making sure you ask for some specifications before you start a project. You won't always get them but you can always ask. Whenever things need to be rewritten you should make a subtle reference to the specifications, even if they don't exist. Sometimes you can be blunt and non-subtle, sometimes you can't. **shrug**

    If the higher ups wonder why things are taking so long be sure to remind them of things that are being done 2 or 3 times (subtle / non-subtle **shrug**). It all counts as work done for the project.

    Sometimes people will figure it out and start planning ahead and sometimes they won't. I wouldn't get too worked up about it because it's sure to happen again.

    mr greywolf
Re: Specify, Specify, Specify (or test, test, test ... and taste)
by stefp (Vicar) on Oct 01, 2001 at 16:02 UTC
    The Perl way (and the extreme programming one BTW) uses more tests than specifications. Specifications are like consultative assemblies, they have no direct enforcement power. They can't have until they are translated into enforcing tests.

    The problem of specifications is the choice of language. Specifications in common language are accesible to everyone including management and customer, but they are ambiguous. Specfications in a more formal language are hard to write and to read.

    In a sense, tests are a form of specification. The big problem is that they are only palatable to programmers.

    -- stefp

      I've seen a couple of mentions of XP in this thread and, while I haven't looked into too deeply, I'm not sure of its utility for certain projects.

      One project that I'm currently working on requires some very difficult database design with products that have widely varying specifications. Books have bindings and a publisher, fishing lures have type of material and the type of fish they're for, clothes have sizes, etc. Because the nature of the products that we get in will change dramatically, creating separate tables for each will not be an option (not to mention that this would violate first normal form and cause us a lot of headaches). As a result, I've "conceptualized" a scheme where each attribute would be in a separate table and I would have a lookup table matching products and attributes. This seems easy, but then we wind up with display problems. How do we organize a variable number of product attributes in an aesthetically pleasing and comprehensible way for the end user. Also, we need to access information regarding the structure of the database so that when we dynamically generate input forms for our dealers, we don't need to worry about them entering a 2000 word description for an item that only takes 255 characters. This seems to imply a lot of careful design work up front, but XP seems to slough that off in favor of tests. I really don't see how one skips designing complicated systems in favor of testing it. How do you manage that?


      Vote for paco!

      Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.

        I've seen a couple of mentions of XP in this thread and, while I haven't looked into too deeply, I'm not sure of its utility for certain projects.

        XP stands here for eXtreme Programming, not eXperience points. :)

        I am no specialist of extreme programming. I have hardly started to read extreme programming explained on safari (they got non O'Reilly books now). XP seems to me another name for Larry's whirlpool. Instead of the waterfall model where stages are sequential, the whirlpool model retrofit in the specifications the experience of "later stages".

        The problem with the whirlpool (alias XP) is that it asks every involved person to closely follow the project in every aspects which is probably incompatible with a marked hierarchy or a customer who don't want to be involved in every stages of program development: "He, we outsource that chore because we dont want do bother with it. Come back when you are done."

        Probably the whirpool model works best in Open Source Project (like parrot). Flexible languages like perl encourage very fast iterations so it is no surprising that Larry came first with the whirlpool metaphor (without the hype of exterme programming).

        To answer you concern: the emphasis on early tests is to get early feeback. It permits to "enforce" the specification -- like I said in the previous node -- or to update it. I don't think the whirlpool model encourages to supress specification. Each iteration involves each of the stages that follow each other in the waterfall model.

        -- stefp

Re: Specify, Specify, Specify
by one4k4 (Hermit) on Oct 01, 2001 at 16:47 UTC
    A very well written post, Ovid. It summs up times I have here at work too. We're working on a payroll/HR based product, and have decided that go-live is sometime around next April. The higher-ups are reporting status to the VPs on a scale of how many "check" marks we have on the to-do list.

    We all know that one thing could lead to another thing that isnt even on the "to-do" list...

    My boss, whom doesnt report to the VPs, said it best: "We're measuring how close we are to completion of this road trip based apon how much gas we've used".

    Another spec thing that bites me are functional specs when dealing with html. If you're supposed to code to specs, when specs havent been written you're in a bind. But when you're supposed to write said specs based apon demo/proof of concept code (think: html layout and such..) you're in a worse bind. Because the business units dont understand the difference between proof-of-conctept and final product.

    Thats why projects take so long to do. The coding is a piece of cake. As long as you dont have any baggage to pull you away from your current "top priority to-do".

    _14k4 - (
Re (tilly) 1: Specify, Specify, Specify
by tilly (Archbishop) on Oct 02, 2001 at 00:26 UTC
    Heresy time.

    I do not consider demands for specifications to be always reasonable, nor do I consider strict adherance to them to always be a good idea.

    Specifications are often a programmer's way of telling the customer to nail down what they want in detail before the customer can possibly really know. And then when the programmer is done the specifications is a weapon that the programmer can wave around to say that people really did ask for the system they got which doesn't do what they want. And finally since users do not understand the language of specifications, it is guaranteed that they won't produce what a programmer needs to see, and the result is that the programmers get a virtually guaranteed place to start complaining from.

    In short demanding a complete formal specification is a seemingly reasonable demand which is doomed to fail in many real-world situations. And users who have been around this path a few times are likely to be wary, they know that programmers (like lawyers) can make weapons of words. If they answer these innocuous questions, they will get what they didn't want and be told that it is what they asked for in the first place. And if it is unreasonable to ask a programmer to make, "Just another small change" with no idea what the implications of that change are, it is likewise unreasonable to ask the user to visualize in vast detail the needs of a system they have never seen, particularly when the demands of the business are likely to have changed in unanticipatable ways by the time it is delivered.

    Instead clemburg above indicates a better approach. Yes, the programmer needs some sort of specification. However if you are going to get one, you are going to need someone who understands specifications to sit down with someone who can recognize what a successful system will be, and they need to iterate the design. The specification should be in a combination of media, pencil drawings, example tables laid out in a spreadsheet, use cases, etc. It may need to make it clear where there are grey areas which will be adjusted later one you have a better sense of what is possible and works. It may need to specify some things as being up to the developer, in which case they truly should be. For any complex project it will be incomplete, and this should be formally recognized with contact numbers, and plans for delivery of prototypes from which the spec can be refined.

    How much detail is needed will depend on the scope of the project. Clearly a new accounting system will require much more process than a form to email script. However some of the traps to look out for are specifications too incomplete for the programmer to work productively, specifications that the user does not understand, irrelevant and limiting details fixed in the specification which it would be better to leave open, and decisions fixed in stone before people can reasonably know what the answer is. Avoiding these traps often steers you in contradictory directions. But then again, if it was easy, they would't need to pay the big bucks...

    Incidentally if someone gives you a spec that you have to make decisions on to implement, either be sure that you have the authority to make those decisions, or else throw the problem back and find something else to work on until you have your answers.

      Couldn't agree more. My $job is to be one of these middle men in the process of making sure everyone is getting what they want. And for sure, whenever the people I work with attempt to bypass me and work directly, in what they call the interest of time, they end up in a mess for sure. You always need someone with one foot in each pond whenever extreme views meet. Layer, interpreter, layer; layer, interpreter, layer...

      We speak the way we breathe. --Fugazi

      Agree with you that customers might sometimes feel trapped by specifications. Many people can visualise old system, and often you do not just reimplement old system, because there was a reason to do it better.

      I found in these cases useful to write my understanding of what specifications are, and sent them to comment, so users do not need to start with blank page. And is easier for them to comment/criticize if somebody else (me) wrote it.

      Also, after writing "draft specs", I can either continue with another project, or analyze design possibilities of this new one, while ball is on the user's side.

      Yes, specs are important. And as they say, "if something is important and you want it to be done correctly - do it yourself".

      To make errors is human. But to make million errors per second, you need a computer.

Re: Specify, Specify, Specify
by perrin (Chancellor) on Oct 01, 2001 at 17:53 UTC
    I'm currently in the middle of a large and poorly specified project, so I feel your pain on this. However, it seems like there has to be a way to properly communicate what's needed in a small project without requiring a formal document that may take a significant amount of time for the customer to write. It also seems like these documents never include what I need on the first try, and I end up having to go back and ask for lots of clarification anyway. Sadly, most of the people who are asked to write the specs for this stuff know absolutely nothing about issues of web UI like confirmation pages and error handling, and even less about tech issues like which templating tool will be used.

    Maybe the answer to this problem lies in one of the lightweight methodologies like Extreme Programming. I don't have any hands-on experience with XP (yet!), so it's hard to tell.

Re: Specify, Specify, Specify
by jlongino (Parson) on Oct 02, 2001 at 04:36 UTC
    I can believe that non-development bozos would give such ludicrous specs but find it hard to understand that your boss actually allowed it or even thought that a project that involves creating/testing a web form would only take 15 minutes (even if you already have templating tools in place).

    The appropriate action to this type of situation is to make a quick note of the unknowns or jot down the specs as you understand them based on the given specs and leave whomever an message saying "I can begin work just as soon as possible but I need some of the following issues resolved: . . ." and continue with another project already underway.

    I'm very lucky to have the boss I do because he protects my time and if I start feeling stressed, he'll sit down with me and help me re-prioritize my tasks. He would never accept those types of specs and would probably thump me on the head if he discovered that I began coding any type of project (small-large) without running the specs by him first.

    Granted some may not have the luxury of the type relationship I have with my boss. He would be the first to tell me: "You made the best decision you could--given the circumstances" and then make sure that the bozos know why they didn't get what they wanted when they wanted it.

    Of course there are sometimes extenuating circumstances (like when the bozo is the Company President) and you just have to suck it up and keep going. But you can usually figure that there will always be higher ups that consider three hours of your time much less important than 15-30 minutes of their own time (the amount of time it might take them to pull together a few reasonable specs).

    "The optimist proclaims that we live in the best of all possible worlds, and the pessimist fears this is true." -- James Branch Cabell