How to write long programms?

[pokemonk]

well i'm working on a cgi programm, and i've noticed it's starting to be pretty long (250), so i'd like to know a way of doing organized programming. like whe i'm writing it i have some great ideas, and they work, but if i look a the code 2 days after i wrote it, i don't understand it. well i hope you guys help me on this one!!

[tilly]

Quite a few areas of computer science are devoted to parts of this question. Every good programmer has read multiple books on it. This is not a simple topic.

You might ask how much could really be said about this. The answer is that you would be astounded.

A few random topics are how you should name variables (chapter 1 of the Camel has some interesting advice for that), how much to indent (anything from 2-4 works as long as you are consistent, Larry Wall uses 4), to how long a function should be (my estimate is that I average about 10 lines, a general rule of thumb is that 50 lines or one screen should be a max), how do you give feedback (it turns out that code reviews are cost effective just in terms of bugs discovered), management techniques, programming methodologies... You name it, it has been studied and frequently useful information is known.

The importance of the topic can be seen in Perl by looking at how many functions exist for little purpose other than to help people keep programs managable. A short list includes bless, local, my, package, require, sub and use. Don't forget the pragmas strict, vars and diagnostics. (And turn warnings on as well!) In addition Perl has quite a bit of syntax (think comments, POD, flexible quoting, etc) that is geared towards letting you say things in a natural way. (There is a school of thought that says that making the gap between how you think about a problem and how you say it small is good. Perl tends this way. There is another that says it is good to always say things in a consistent way. The truth is somewhere in the middle, it depends on the characteristics of the product and developers whether flexibility hurts or helps.) Getting familiar with and using those tools on a regular basis will go a long ways towards helping keep your script managable.

But now I should reveal a dirty little secret.

We do all this, and we still fail. Individually and as a whole, the programming industry routinely fails in both insignificant and spectacular ways. The central problem of programming is the same today as it was 50 years ago. Namely keeping in control of the complexity of software. There is no need to be cynical about research into this problem, we have learned a lot over the years, and we can handle much harder problems now. Your script is quite managable. But still our basic barrier is in our ability to keep track of what the computer has been told to do. We can move where that barrier is, but not stop it from being a problem in the end.

From of this research there is a key insight that comes up again and again. That is the value of modularity. You really want to take problems, and break them up into smaller problems, then produce solutions to each smaller problem, and put the components back together. It is important that each component be independent of the others. This is true whether your components are functions, modules, etc. As a random instance Dominus has a good rant about why this matters. Indeed most of the functions and pragmas that I listed above in some way serve to make it possible to isolate components, or to help you verify that things have been properly separated.

If you want good places to pick up more advice on this topic, tombstone 163 gives links for a number of books, including both Code Complete and The Pragmatic Programmer. Most good Perl books will give practical demonstrations of how to do it. And, of course, look around at this site. There is a lot of good advice, and many monks offer links to even more. I would personally recommend visiting the home nodes for Dominus, merlyn and chromatic for a start.

(Next up, how to write a long post without meaning to!)

UPDATE
Edited to fix minor grammar mistakes.

[japhy]

Writing code you don't understand two days later is a bad thing. Perhaps you need to comment the code more.

The goal in this case, though, is to learn how to modularize your code effectively. Most CGI programs I write use a dispatch table:

my $thing = $cgi->param('action');

my %actions = (
  login => [ \&login ],
  main => [ \&view_page, 'front' ],
  index => [ \&view_page, 'index' ],
  unknown => [ sub { die "unknown action '$thing'" } ],
);

my ($func,@args) = @{ $actions{$thing} or $actions{unknown} };

$func->(@args);

Basically, I set up a hash of function references (and their arguments). Then I call the function corresponding to the action.

Most of my code is broken down into functions in the case of main functionality and repeated functionality -- in other words, the main actions (logging in, viewing a page, searching) are in functions, and common tasks (reading a config file, saving data) are in functions as well.

japhy -- Perl and Regex Hacker

[CiceroLove]

For what it is worth, I tend to code my CGIs as functions. What I mean by that is that the 'commands' are typically only about three or four i.e.,


&parseArgs(@argArray);
&manipulateData(@returnArray);
&cleanUp;

My reasons for this are simple. By making a lot of subroutines, I can be coding a part of the overall CGI without constantly having to wade through lines of code I don't want to touch. It allows me to think in portions of the CGI and think in the flow of data overall and then get down to the nitty gritty. And of course, I notate copiously. My first CGI I commented on every line. I don't do that anymore but people always marvel at the fact that typically my comments are a fourth of the text in the whole script. They laugh but I always am able to return and fix stuff without first having to figure out what I was thinking when I did x function.

Hope that helps you some.

CiceroLove

Fates! We will know your pleasures: That we shall die, we know; 'Tis but the time, and drawing days out, that men stand upon. - Act III,I, Julius Caesar

[Jouke]

When code starts growing, it can be a good idea to split things up into modules. Give your variables sensible names. Same goes for subroutines.

More importantly: document what you do. If you write code of which you even think it has the slightest chance that someone else will not immediately understand what happens, write a commentline in which you describe what it does. That does not only go for complicated regexes, but also for calculations and stuff like that.

Hope it helps...

Jouke Visser, Perl 'Adept'

[dws]

... but if i look a the code 2 days after i wrote it, i don't understand it.

There's some very good advice above, with one caveat: we're not inside of your head, so we can only guess at why it is that you're having difficulties understanding your own code.

Solving this is going to require some introspection on your part. You're going to need to take a step that may be difficult: Each time you find yourself not understanding your code, step back for a moment and try to get more specific about what you're not understanding. Are you lost in the the structure? Are you lost trying to figure out what your regular expressions are doing? Are you lost trying to tell where things happen in your code? Are you lost trying to figure out why things happen in your code?

Once you've labeled a particular problem, a good step is to consider what you could have done up front (2 days earlier) to avoid that problem. This may using clearer variable names, or whitespace around certain syntactic elements, or more (or less) commentary. Without seeing your code through your eyes, it's hard for us to guess what you'll need to do.

The next step is an important one: Write your observations down. Keep a log. With a written record, you're more liable to see repeated patterns that you miss in day-to-day coding and debugging. And the mere act of writing problems down can be a creative stimulation.

You may find that you only need the log for a while. But if you start working on larger projects you'll run into a different class of problems, and you might want to start keeping a log again. A 250 line CGI script might seem large now, but one day you might find yourself working inside of a 250,000 line system.

[Masem]

Another way to do it, besides those offered, is if you are working on a CGI that is to run a whole site (as opposed to one specific function) which has many different functions is to break down the cgi into multiple scripts, and create a module that combines any common variables and functions into one place; this works much better when you have mod_perl running than without since there might be a lot of overhead calling in this CGI for one sub-script, but not when it's kept in memory. Generally the hardest part here is maintaining consistent HTML output in terms of everything outside of the specific CGI script for each sub-script, but the use of templating classes (HTML::Template, Template Toolkit 2) makes this part easy.

---

Dr. Michael K. Neylon - [e-mail]
"You've left the lens cap of your mind on again, Pinky" - The Brain

[jynx]

People have already written that one should comment, but i would like to make an addendum to that:
Comment well.

Bad comments are worse than no comments at all. Some areas need heavy commenting because they do the black magic of the program. In these cases, it's usually better to type up a well-written paragrah (in full english) that explains why the section does what it does and how. Then post that paragraph above the black magic and make sure it's noticeable and easy to read (using various commenting styles to make it obviously important)

As a general rule of thumb, if you can't comment it well you might reconsider how you're approaching the problem that the code solves. i've had to reinvent a code section three or four times on one program just because i couldn't describe why i did it that way that made any sense.

And lastly, to make sure that beautiful paragrah makes sense, see if a co-worker or pedestrian (hopefully with no knowledge of that section fo the program) can read and understand what you're trying to do. If they can, then when you go back and read it 2 days later, you should also have no problems.

This is just my experience, YMMV, HTH,
jynx

PS If it takes more than a paragraph to describe some code, maybe you're doing too much with that section, it might needs be split up, maybe...

[mothra]

well i'm working on a cgi programm, and i've noticed it's starting to be pretty long (250)...

250 lines...long...? Wow. :) I yearn for the project I get to maintain that is any LESS than 10,000 - 15,000 lines (which I would still call a fairly small program, certainly not a "large" one :).

But anyways, for me, I find that the OO paradigm is a nice way to manage complexity. Taking the time to think about pieces of functionality as "objects" within a "system" (notice the Dr. Evilish quotations there) is the most natural way for me to manage complexity, and can lead to some very elegant solutions (not that functional programming can't, of course). There's no magic answer though, because everyone does things in their own way, and programming is hard. Think about what you're doing, seeing how you can break up your source files into logical units (aka, modules).

• Ideally, you'll want to follow what tilly would recommend and keep your functions to <= 50 lines. In practice, it's almost impossible to always stay under this benchmark all of the time, but it will remind you to NEVER fear making your code modular. ie. Even in your 250 line program, your code may be such that it might require 15 functions to keep the cohesion high, and the coupling low.
• Never hard code magic numbers, or you'll experience magical side-effects as your system grows.
• Constantly refactor.
• Avoid global variables because again you'll experience side-effects as your system grows.
• Don't overcomment. Too much commenting will get in the way of what could otherwise have been a nice, compact piece of code.
• If you're really concerned in terms of "how do I organize my files?" when working on larger systems, use an IDE or text editor that supports "projects".

Needless to say, this is a question that is hard to answer, because again, everybody does things in their own way. The more code you write, and the more time you spend thinking about the way the pieces of your system are working together (or aren't), the sooner you'll figure out what works for you.

[ybiC]

Your fourth suggestion (Avoid global variables) struck a chord for me, mothra, as I've recently received that same advice for improving the quality of my own modest Perl scripts.

What negative side effects result from using global variables in large programs?
    cheers,
    Don
    bumbling toward Perl Adept
    (it's pronounced "why-bick")

[gopher]

I just started reading a book on C++, and the entire first chapter is about what you seem to be having problems with. The basic overview is to use meaningful variable names, make your code into functions, and heavily comment your code. No one wants unreadable code, especially when they wrote it themselves.

"Mr. Zoothornrollo, hit that long lunar note, and let it float."

[scottstef]

I am not a programmer by any stretch, a psuedo shell scripter at best.
I look at writing long code like the old saying "how do you eat an entire cow?" - one steak at a time.

I always break the code into smaller parts and then work on one part at a time until i get each part to work correctly. After I get one section of code to work the way i want, I start the next. After that one is done, i try to tie the two together. If they work correctly I go to the third and so on..

Just my thoughts