http://www.perlmonks.org?node_id=50489

belize has asked for the wisdom of the Perl Monks concerning the following question:

I have searched perlMonks with little success on techniques, software, suggestions on how to go about planning your software prior to writing any code.

I know that we use the old "Flow Chart" method through several revisions. We refer to this Flow Chart a number of times, and will even develop specific ones for parts of a subroutine.

I would also think that these flow charts would be invaluable when trying to understand what others are doing, or going back after sometime and trying to under stand what YOU did.

Any comments, ideas, suggestions on the most efficient and "correct" way to plan or map out software prior to writing? Anyone use software to plan (i.e. Visio).

BTW, in all the Open Source code that I have looked at, I have never seen a flow chart or map of the logical flow of the software.

Replies are listed 'Best First'.
Re: Planning your software before writing
by chromatic (Archbishop) on Jan 08, 2001 at 21:39 UTC
    Here's my take on things, (follow the link in neophyte's post for better reasoning and more cogent analysis):
    • Software changes all the time
    • Software requirements change all the time
    • Doing a complete design up front is a big investment that pays less and less the more you change
    • You can't predict what will happen more than a week or two in advance
    • When you actually start coding, you'll find better ideas
    • No one wants to update the design documentation while you're coding
    • A large design at the start can lead you to the trap of making things too flexible, adding functionality you don't need just yet
    • If the project is always in a functional state, you can deliver it to the customer nearly anytime
    • If the customer chooses the features that provide the most business value and you work on them, you can maximize your return on investment
    • Write the simplest code that could possibly work. It's easier to debug and to understand that way.
    I'm not saying that design isn't important. I'm just saying that spending a couple of weeks at the start of a project doing nothing but design is not (in my experience) going to pay off in six months, when the specifications have changed, the original design didn't work, and no one wants to update a few dozen CASE diagrams or pages of UML.

    Certain industries where accuracy and reliability need to be mathematically proven may have different requirements.

    XP practitioners try to 'code by intent', where they write a test case for the feature they're about to add, let the test fail, then write code to make the test pass. That's design in the small.

    For design in the large, they write stories about the features the software needs. The customer arranges them by their value in the shipped product, and the team tackles them in that order. Design is done as-you-go, with a little up front investment in the basic architecture (the simplest that could possibly work), breaking a story down into programmer-afternoon-sized-tasks, and continually refactoring.

    That may or may not work for you, and it may or may not work for my latest project. We'll see.

      OK hows about this... Design is important cause software engineering is hard. Gifted engineers are able to code with little written design as soon as other engineers are added then design becomes very important. Or to put it another way if you have one programmer who is dead smart then he (or she) may be able to skip the design phase and still produce a damn fine project. As the project grows in size the requirement for design rises exponentially. XP programming simply stated avoids the issues because it quite rightly says that design is hard so we wont do it. You have correctly explained why design is hard, all of your points are valid.. However it is important to realise the cost fractionalising the design, the benifit is easy to see. Picking a point at random: software requirements change all the time so in a traditional environment we will have to change the design and the code that takes lots of time. In our XP world we have no design to update so we only have to update the code (although I have written this flippantly I do not mean it be so this is a genuine reason why XP is good) The problem is you have no design so each change becomes a little harder to implemnt as the impact is harder to see because there is no big picture, because there is no design.

      Formal design is required for the following (off the top of my head) reasons

      • To allow agreement between developers upon interfaces and functionality. A large project will have many elements all ready to collide the design will ensure that they all use the same messaging protocol and so on.
      • To provide a road map for developers to be able to explore code they did not write or were not invloved within (or have simply forgotten). No matter how good a developer is he (or she) will get up to speed more rapidly with a design document that code. Reverse engineering is hard (trust me that is what I am doing now)
      • To allow for impact analysis to take place regarding changes to core processing.
      • To provide a common understanding of where you are trying to get to before you arrive.
      • ... and so on and so on
      I have looked at XP with interest but never tried it. My impressions are all negative please let me know how you get on with it.. I would love to be wrong and for XP to work.. twould certainly make our lives easier.
      --

      Zigster
        Those are common criticisms of XP, and they're very understandable. I may not have explained it correctly, so I'll take a shot at clarifying. (I also don't mean to say that XP is a panacaea or suitable for everything. I'm just trying it with a couple of my projects, and it's working pretty well.)

        It's not that design is hard, though it is, it's that up front design is very hard. That's why projects like the Gimp, Apache, Mozilla, and Slashcode are all facing fundamental rewrites. They invested some time in design before they started coding, but the uses and needs of their software grew beyond their original design. They designed themselves into a corner.

        XP may or may not completely alleviate that -- it depends on your team's abilities. Instead, it says, "Design is good. Let's design all the time."

        To address more of belize's post below, I'd take something like the Ultimate Bulletin Board, and write stories like the following:

        • Users must be able to post comments
        • Users should be able to see all comments arranged chronologically
        • Comments should strip out dangerous HTML
        • Moderators/administrators should be able to edit comments
        • Moderators/administrators should be able to delete comments
        • Users should be able to mark up their comments with smilies, bold, underlines, and italics
        • Users should be able to edit their comments
        • Moderators/administrators should be able to add their own markup
        At some point in the future, you'll probably want to use a database for this, probably around the time you hit story 5. Are you going to bank on your ability to design the right db schema when you hit story two? XP would say that you should do as much as you can *without* adding that kind of infrastructure until you absolutely need to. Especially knowing that the requirements will change.

        I, for one, would rather spend my resources completing the first few stories than building a database, arranging the tables just so, then having to modify the tables and convert the code four or five times. I'll probably have to do it once or twice, but the simpler the existing code and design by the time it is absolutely necessary to add the database, the less chaos there will be.

        It's not a case of no design vs. full design, it's a case of design only as much as you need for the next two weeks vs. design everything.

        One last what if... what if you spent a week designing the interface to a billing system your customer might want in a year, then he tells you that they have decided to use an existing front-end and want you to work on a batchloader? (Design what you need right now.)

        http://extremeperl.org is an excellent introduction to XP, especially with a Perl bent.

        Being right, does not endow the right to be rude; politeness costs nothing.
        Being unknowing, is not the same as being stupid.
        Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
        Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

      It is interesting that most of ther responses seem to assume a very large project. But a small company like ours deals with very small applications with very specific uses. Maybe what is needed is a guage of "small" versus "large".

      How would you rank a project such as Infopop's "Ultimate Bulletin Board" which appears to be very successful and continually growing.

      Or John Cokos "Hyperseek" search engine which has grown over the lsat few years.

      If you are not familiar with either one:

      http://www.infopop.com

      http://www.interactive-web.net

      Both of these appear to be small applications. But would require some heavy duty planning and documentation to maintain.

      With respect to the stories, the method I've been taught is that your programming team typically works on a two week cycle.

      Every two weeks you have have a meeting with the customer. At this meeting you go over what has been completed. The customer then hands over the next set of stories and you only accept the amount of work you know you can do in two weeks.

      In this manner you never have all the stories, you only work in the here and now.

      There are more things to be taken into account here, but I won't bother with them at the moment...

      Cheers!

Re: Planning your software before writing
by KM (Priest) on Jan 08, 2001 at 19:41 UTC
    I have used Visio before, but tend to go back to the analog whiteboard. I have often thought of a 'correct' way to plan and map software (at least Perl-centric software). I have worked a few places in my time, and find that a 'correct' way one place wouldn't work somewhere else. It all depends on the people on the team, company policy, etc... In general, however, I think there are a few things that always help, and are staples for planning software.

    • Clear 1000 foot view of software. It will do X, and be used by Y.
    • Map of any integrations. Interfacing with existing database? Need new database? Need new tables? Interfacing with software? Is there an API? etc...
    • General bubble chart (flow-chart without real flow) of 500 foot view. Can this be modular? Can sub-teams work on various parts (ie. Admin section, backend, front end, UI, etc...).
    • SPECS!!! SPECS!! SPECS!! The specs should have specifics and details of how things will work. These should be worked on by management AND development.
    • Final specs should include list of technologies being used (XML,Perl, Apache, Oracle, etc...) so they can be installed.
    • Now, have fun on the whiteboards. I see things work well when sub-teams create DETAILED flows of their bits, then explain their bits to other sub-teams. This helps you see how the bits will touch and mesh and allow for people to know what API's to make sure they have for other bits. Then, revise flows and details as needed.
    • Develop! Changes will be made, things will be added/removed.. there is no permanence in software design.

    There is a lot in between (I could probably do a book on dos and do-nots of real-world software design), but those are some highlights I have seen work well. Generally I don't like to use any flowchart software (unless someone else will maintain the flowcharts) because whiteboards are better (if you are luckly, you have a whiteboard that prints). I love walking into a room and seeing all the whiteboards marked up with the flow of a big piece of software!

    Cheers,
    KM

      I see your point. Overview to specifics and if the specifics can be broken into teams the better.

      I am coming from a little smaller company (9 employees) that are struggling to fill the niche of customers that have around 10-15k to spend instead of the 800k or more for the big boys.

      I have often wondered whether it is better to built something form scratch or to look for something that has already been done and modify it. The first way takes more time, but gives you exactly what you want. The other takes less time, but often does not give you what you want.

      Programming always seems to be a trade off between time, money,and functionality (maybe we should add reliability?).

        The other takes less time, but often does not give you what you want.

        Well, that depends on what you find to build on. If you find something (or some things) that were well designed to build off of, it can save you time. If you download crap to use, then you have crap to work with. I once had to do a calendaring system.. I found a Perl one on Sourceforge (don't recall the name right now) and it was decently written, and had much of the functionality I needed, and was easily expandable (I added tons of things, and now use my version for my own personal use). So, in that case it saved time. Before finding that one I looked at another, which was crap, and I saw a long hard road ahead. I didn't start from scratch because of my task load, and I happened to find something very usable.

        Programming always seems to be a trade off between time, money,and functionality (maybe we should add reliability?).

        I say, let the managers worry about money (and maybe the time) and the programmers worry about functionality (reliability, readability, scalability, and all other bilities). If you happen to have a boss who isn't afraid to tell a customer 'it will take an extra two weeks, because we want to make sure it is a good piece of software now, rather than fix things later', all the better. Crap gets created when you have someone saying 'Well, just make it work they want it on Friday'. Personally, I think a programmer should say 'No, it will be done when it is right', but some value the paycheck to much :)

        Anyways, when you have the 1000 foot view, it is a good time to research what is available that can be a base to work from, IMO.

        Cheers,
        KM

Re: Planning your software before writing
by merlyn (Sage) on Jan 08, 2001 at 20:39 UTC
    I'm a sort a "start at the top and bottom and meet in the middle" guy. I think about the top level algorithm first, coding it in understandable chunks, calling stubs for most of the functionality.

    Then I look at code reuse, and figure out what bottom chunks are already written for me, and start writing glue in between the top level and bottom level. If I'm missing low-level technology, I code that, trying to be general enough to solve this problem as well as the next couple of predicted revisions or reuses. I've written a lot of code in my life, so predicting reuse is pretty second nature. {grin}

    I run my program after adding about every 10 lines of code or so, stopping to insert print statements if it doesn't run right. I never have to look at more than 5 or 10 lines to debug then. The program becomes its own test harness.

    -- Randal L. Schwartz, Perl hacker

      This is the way I have been programming also and have found it awful when it comes to upgrading, modifying, trouble-shooting, especially a year or so after implementing. There seems to be an inordinate ampount of time spent trying to figure out what I've done.

      I think this method might work well with small applications, but as move on to bigger things on the net, it appears to reach a limit of usefulness.

      Of course I am not in the same league as merlyn when it comes to programming perl.

Re: Planning your software before writing
by zigster (Hermit) on Jan 08, 2001 at 20:43 UTC

    Hay ho one can of worms opened ;-) Software design is a very passionate argument and one that I cannot do justice to here so I will simply point you at the two techniques I use most frequently.

    • JSP for modeling small sections
    • UML for system wide modeling

    Jackson Structured Programming (JSP), a mature yet simple (and so useful) method for showing program flow. I use this to produce a road map for subsections of an application. I believe it is talked about in code complete McConnell (A good book to have, even if it does not cover JSP, I am -pretty- sure it does) I had a quick look and found a cobol esq view of JSP Although it is cobol based it does describe the method well.

    UML, this is in vogue atm as an OO modeling language, however as I have said before OO is only a way to understand a problem. There is no reason an OO design -has- to yeild an OO solution. As you mention flow charts I have assumed that you are writting nonOO code. Do not eliminate UML as a modeling language you will not regret learning it. UML allows whole system modeling. It defines a number of diagrams from use case (that allow you to capture and present user requirements) to sequence diagrams that show the calling flow (they model a similar attribute as JSP). Take a look at For UML related resources . There are CASE tools to help you write UML Together , to be honest not appropriate for perl projects it is slow and tightly coupled to java/C++ a better option would be dia a gtk tool that really rocks.

    If you want more references or more specific suggestsions then post again I have lots and lots ;-)
    --

    Zigster

Re: Planning your software before writing
by clemburg (Curate) on Jan 08, 2001 at 23:34 UTC

    You will want to read Code Complete by Steve McConnell.

    One simple trick that this book taught me is to write a high-level, natural-language description of your task, comment it out, and write the code corresponding to each modular part of the description. This way, you even get comments for free.

    Another good thing is to acquire a feeling for when top-down styles (structured programming, most OO stuff, flow-charts, etc.) and bottom-up styles (interactive environments, rapid prototyping, tool development) of programming are appropriate. Unfortunately, this is very hard. Since most of today's techniques are top-down style, I think it is good practice to learn a functional programming language (Scheme, Lisp, Haskell, etc.), since these tend to encourage bottom-up style.

    Christian Lemburg
    Brainbench MVP for Perl
    http://www.brainbench.com

Re: Planning your software before writing
by neophyte (Curate) on Jan 08, 2001 at 21:16 UTC
    I just wanted to give a link that I picked up elsewhere on this site:

    XtremeProgramming gives lots of information about the Xtreme Programming technique, complete with examples.

    neophyte

Re: Planning your software before writing
by extremely (Priest) on Jan 08, 2001 at 19:59 UTC
    After I get a big overhead view, I worry most about data flow and storage. I work over APIs for flaws and DBs for nastiness that will make my life harder. My flow charts are mostly data intersections where I track what data needs to move where. Once I have that, modules, structures, and objects just seem to appear around the chunks =)

    --
    $you = new YOU;
    honk() if $you->love(perl)

Re: Planning your software before writing
by blueAdept (Beadle) on Jan 08, 2001 at 20:38 UTC
    Flowcharts are good, but whats much more valueable to me is documentation of the data structures, and a heirarchial diagram of how you implemented the processes of the flow diagram. Documenting the objects is very valuable(both data and the code itself - if its object oriented).

    Flowcharts alone don't go very far with me, they leave a lot unsaid..up to your interpretation/guesswork. If all you have is the flow diagram, you need to delineate where/how the various tasks in the flow are implemented in the coded routines.
      " much more valueable to me is documentation of the data structures"

      Is this documentation of data structres done within the code, or do you make it a separate document? Wouldn't it best to develop a system where flow charts and data structures flow from one document attached to the software?

      I am just surprised that I have never seen any of this with any Open Source software that I have looked at.

        In the business world it'd all go into a separate document. For a script I'd write to send out for other's to use on the net, it'd probably go in the code. In most cases though the documentation is long enough(and extensive) that it'd seem a waste of real estate to put it into the code as comments/pod IMHO. Its also more readable(and thus understandable) for me when its in word processor with a nice clear(slightly larger) font and pretty formatting.

        I've also worked with scripts written by others that were so heavily commented that it would have been easier to understand without any comments at all. Everyone is slightly religious about how the prep work/documentation of their application is done..