Troubleshooting Perl CGI scripts

[brian_d_foy]

This resource is intended as a general framework for working through problems with CGI scripts. It is not a complete guide to every problem that you may encounter, nor a tutorial on bug squashing. It is just the culmination of my experience debugging CGI scripts for ten years. This page seems to have had many different homes, and I seem to forget it exists, so I'm adding it to the Monastery's canon. You can send any comments or suggestions to me at [e-mail]. You may also want to see the CGI Meta FAQ for a list of references.

---

Are you using Perl's built in features to help you find problems?

If you have not turned on warnings with the -w switch and are not using strict, you should be. These things help you find problems. You may need a bit of time to get used to them, but they save you many hours of time. You need them to go through this troubleshooting procedure.

Did you output a valid CGI header first?

The server is expecting the first output from a CGI script to be the CGI header. Typically that might be as simple as print "Content-type: text/plain\n\n"; or with CGI.pm and its derivatives,

print
header()

. Some servers are sensitive to error output (on STDERR) showing up before standard output (on STDOUT).

What did the error log say?

Servers keep error logs (or they should, at least). Error output from the server and from your script should show up there. Find the error log and see what it says. There isn't a standard place for log files. Look in the server configuration for their location, or ask the server admin. You can also use tools such as CGI::Carp to keep your own log files.

What are the scripts permissions?

If you see errors like "Permission denied" or "Method not implemented", it probably means that your script is not readable and executable by the web server user. On flavors of Unix, changing the mode to 755 is recommended: chmod 755 filename. Never set a mode to 777!

Are you using use strict?

Remember that Perl automatically creates variables when you first use them. This is a feature, but sometimes can cause bugs if you mistype a variable name. The pragma use strict will help you find those sorts of errors. It's annoying until you get used to it, but your programming will improve significantly after awhile and you will be free to make different mistakes.

Does the script compile?

You can check for compilation errors by using the -c switch. Concentrate on the first errors reported. Rinse, repeat. If you are getting really strange errors, check to ensure that your script has the right line endings. If you FTP in binary mode, checkout from CVS, or something else that does not handle line end translation, the web server may see your script as one big line. Transfer Perl scripts in ASCII mode.

Does the script give you warnings?

You can check for warnings by using the -w switch which I recommend you use during all development. Warnings are fully explained in the perldiag man page, or by using the pragma use diagnostics;. If you make your script "-w clean" you will have less trouble with it in the future.

Is the script complaining about insecure dependencies?

If your script complains about insecure dependencies, you are probably using the -T switch to turn on taint mode, which is a good thing since it keeps you have passing unchecked data to the shell. If it is complaining it is doing its job to help us write more secure scripts. Any data originating from outside of the program (i.e. the environment) is considered tainted. Environment variables such as PATH and LD_LIBRARY_PATH are particularly troublesome. You have to set these to a safe value or unset them completely, as I recommend. You should be using absolute paths anyway. If taint checking complains about something else, make sure that you have untainted the data. See the perlsec man page for details.

What happens when you run it from the command line?

Does the script output what you expect when run from the command line? Is the header output first, followed by a blank line? Remember that STDERR may be merged with STDOUT if you are on a terminal (e.g. an interactive session), and due to buffering may show up in a jumbled order. Turn on Perl's autoflush feature by setting $| to a true value. Typically you might see $|++; in CGI programs. Once set, every print and write will immediately go to the output rather than being buffered. You have to set this for each filehandle. Use select to change the default filehandle, like so:

$|++;                            #sets $| for STDOUT
$old_handle = select( STDERR );  #change to STDERR
$|++;                            #sets $| for STDERR
select( $old_handle );           #change back to STDOUT

Either way, the first thing output should be the CGI header followed by a blank line.

What happens when you run it from the command line with a CGI-like environment?

The web server environment is usually a lot more limited than your command line environment, and has extra information about the request. If your script runs fine from the command line, you might try simulating a web server environment. If the problem appears, you have an environment problem.

Unset or remove these variables

• PATH
• LD_LIBRARY_PATH
• all ORACLE_* variables

Set these variables

• REQUEST_METHOD (set to GET, HEAD, or POST as appropriate)
• SERVER_PORT (set to 80, usually)
• REMOTE_USER (if you are doing protected access stuff)

Recent versions of CGI.pm ( > 2.75 ) require the -debug flag to get the old (useful) behavior, so you might have to add it to your CGI.pm imports.

    use CGI qw(-debug)

Are you using die() or warn()?

Those functions print to STDERR unless you have redefined them. They don't output a CGI header, either. You can get the same functionality with packages such as CGI::Carp.

What happens after you clear the browser cache?

If you think your script is doing the right thing, and when you perform the request manually you get the right output, the browser might be the culprit. Clear the cache and set the cache size to zero while testing. Remember that some browsers are really stupid and won't actually reload new content even though you tell it to do so. This is especially prevalent in cases where the URL path is the same, but the content changes (e.g. dynamic images).

Is the script where you think it is?

The file system path to a script is not necessarily directly related to the URL path to the script. Make sure you have the right directory, even if you have to write a short test script to test this. Furthermore, are you sure that you are modifying the correct file? If you don't see any effect with your changes, you might be modifying a different file, or uploading a file to the wrong place. (This is, by the way, my most frequent cause of such trouble ;)

Are you using CGI.pm, or a derivative of it?

If your problem is related to parsing the CGI input and you aren't using a widely tested module like CGI.pm, CGI::Request, or CGI::Lite, use the module and get on with life. CGI.pm has a cgi-lib.pl compatibility mode which can help you solve input problems due to older CGI parser implementations.

Did you use absolute paths?

If you are running external commands with system(), back ticks, or other IPC facilities, you should use an absolute path to the external program. Not only do you know exactly what you are running, but you avoid some security problems as well. If you are opening files for either reading or writing, use an absolute path. The CGI script may have a different idea about the current directory than you do. Alternatively, you can do an explicit chdir() to put you in the right place.

Did you check your return values?

Most Perl functions will tell you if they worked or not and will set $! on failure. Did you check the return value and examine $! for error messages? Did you check $@ if you were using eval?

Which version of Perl are you using?

The latest stable version of Perl is 5.8.3. Are you using an older version? Different versions of Perl may have different ideas of warnings.

Which web server are you using?

Different servers may act differently in the same situation. The same server product may act differently with different configurations. Include as much of this information as you can in any request for help.

Did you check the server documentation?

Serious CGI programmers should know as much about the server as possible - including not only the server features and behavior, but also the local configuration. The documentation for your server might not be available to you if you are using a commercial product. Otherwise, the documentation should be on your server. If it isn't, look for it on the web. Consult the CGI Meta FAQ for a list of references.

Did you search the archives of comp.infosystems.www.authoring.cgi?

It's likely that someone has had your problem before, and that someone (possibly me) has answered it in this newsgroup. Archives are available from Google

Can you reproduce the problem with a short test script?

In large systems, it may be difficult to track down a bug since so many things are happening. Try to reproduce the problem behavior with the shortest possible script. Knowing the problem is most of the fix. This may be certainly time-consuming, but you haven't found the problem yet and you're running out of options. :)

Did you decide to go see a movie?

Seriously. Sometimes we can get so wrapped up in the problem that we develop "perceptual narrowing" (tunnel vision). Taking a break, getting a cup of coffee, or blasting some bad guys in Quake might give you the fresh perspective that you need to re-approach the problem.

Have you vocalized the problem?

Seriously again. Sometimes explaining the problem aloud leads us to our own answers. Talk to the penguin (plush toy) because your co-workers aren't listening. If you are interested in this as a serious debugging tool (and I do recommend it if you haven't found the problem by now), you might also like to read The Psychology of Computer Programming.

Did you post an informative and well-developed question to comp.infosystems.www.authoring.cgi?

Once you have gone through all of these steps, you should have all of the information to ask a question that can elicit a good answer. Post in as much detail as possible, and include the smallest bit of code that reproduces the problem.

Edited by davido: Fixed HTML and formatting to eliminate horizontal scrolling in IE.

[bradcathey]

Excellent "tutorial!" I'm racking my brain and the only thing I can add is to use Data::Dumper. And though it's better to use the Debugger, I resort to print statements for quick traces. Thanks for your hard work.

[rinceWind]

You need to add the shebang line to your checklist. Apache (for one) will refuse to run a script without a valid shebang line. It probably belongs next to "What are the scripts permissions?".

On the subject of shebang lines, Devel::ptkdb is worth a mention for debugging. You can get this invoked directly from the shebang line

#!/usr/local/bin/perl -d:ptkdb

On a windows box, a ptkdb window will pop up when you access the url. On Unix, if you configure $ENV{DISPLAY} in apache you can get the ptkdb window to pop up in an X window.

[itub]

I think the latest stable perl is now 5.8.5.

In "What are the scripts permissions?", I might add that sometimes you need to worry about the permissions of all directories up to the one that contains the script. This will be system-dependent, but I've had the following problem when using Apache with suexec on RedHat Linux: the way users and bashrc are setup by default, users have their own group and a umask of 002. That is, when I create a directory, it looks like this:

drwxrwxr-x   42 itub     itub         4096 Jul 30 15:59 test

Even if it has mode 0775, only I can read it because only I belong to my group. However, Apache's suexec doesn't know that and thinks that this is a security risk because others might be able to modify the script. I have to make all directories (e.g., /home, /home/itub, /home/itub/public_html, ...) have permissions 0755 or 0711 or similar.

[pbeckingham]

Very nice Brian. I sincerely hope that you are writing a book.

[Anonymous Monk]

CGI Help Guide

[danmcb]

That's a pretty good list.

I would add some things that I have also found helpful :

1. If you are working on a script of any complexity, modularise. Split functionality out into other modules that don't depend on CGI environment, the CGI itself just instantiates an object, passes it the CGI object, and calls "print" (for example) on that object to produce the output. The great advantage of this is that you can now run a seperate test script on your module (using Test::Simple for instance) and get the complex stuff working right, method by method before worrying about the CGI part. (Some people like fancy names for this kind of thing, I quite like "common sense".)

(This may also have an advantage that moving to mod_perl later (if you want to) will be easier. So I'm told. Don't know yet because I haven't gotten there ...)

2. If you have a lot of HTML, use a template package like Mason or FastTemplate. Get the HTML out of your code and life will start to get easier. You will also reduce the number of bugs you bring in due to making small changes in the way your application looks.

[JaredF]

brian d foy
Nice tips. You might enjoy an old book, still available used: A programmer's introduction to debugging C (Unknown Binding) by Robert Ward. Also, if you can get your hands on it, see Freeman, J.T., Riedl, T.R., and Ward, R. (1994). Debugging software systems. The Wiley Encyclopedia of Software Engineering. NY: Wiley.
Best, JaredF