Beefy Boxes and Bandwidth Generously Provided by pair Networks DiBona
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

CGI Help Guide

by tachyon (Chancellor)
on May 09, 2002 at 05:11 UTC ( #165253=perltutorial: print w/ replies, xml ) Need Help??

This node is being developed to help you get your CGI script working. It is not a Perl tutorial and focuses on CGI issues. It owes much to Tom Christiansen's terse but informative The Idiot's Guide to Solving Perl CGI Problems. I recommend you check this out if you can not find the answers you need here.

What follows is a sort of checklist to help you solve the problem quickly yourself. It is divided into four parts:

A. Before uploading to the Server

B. Uploading to the Server

C. Setting Up on the Server

D. Debugging

 

A: Before Uploading to the Server

1. Does your script compile?

Like any Perl program a CGI script needs to successfully compile before it will run. Until it does this you can forget everything else. To test this you need a command prompt and a working Perl interpreter. If you have *nix you are probably already there. If you have DOS/Windows or a Mac you will probably need to install Perl. You can get a binary copy of Perl, ready to install, for almost any platform here. If you want to compile the source yourself go here.

Once you have a copy of perl you can access from a command prompt ( C:\> in this example under Windows) type:

C:\>perl -c myscript.pl

What this does is compile (but not run) your script. You should see this:

C:\>perl -c myscript.pl
myscript.pl syntax OK

Here are some common errors and solutions:

No Perl!

C:\>perl -c test.pl
Bad command or file name
C:\>

This is telling you that your operating system can not find the Perl executable. Either it is not installed and/or it is not included in your path environment variable. So install perl and either include it in your path or type the full path to the executable. How you find perl depends on your system. On *nix type 'which perl' at the command prompt to get the path (usually /usr/bin/perl). On Windows use Start|Find|perl.exe to get the path (probably C:\PERL\BIN).

A bit of background: the PATH environment variable is a list of directories separated by colons (*nix) or semicolons (Dos/Win). When you type a command name without giving an explicit path your shell searches each directory in the PATH list in order, looking for an executable file by that name, and the shell will run the first matching program it finds.

To check the path on Win32 use:

C:\>PATH
C:\;C:\WINDOWS;C:\WINDOWS\COMMAND;
C:\>

To add to the path use:

C:\>C:\PATH %PATH%C:\PERL\BIN;
C:\>

To check the path on *nix use:

$ echo $PATH
/bin:/usr/bin:

For csh users use something like:
setenv PATH :/bin:/usr/bin
For sh or ksh users:
PATH=:/bin:/usr/bin export PATH

Syntax Errors

C:\>perl -c test.pl
String found where operator expected at test.pl line 1, 
near "printt "Hello World!""
(Do you need to predeclare printt?)
syntax error at test.pl line 1, near "printt "Hello World!""
test.pl had compilation errors.
C:\>

This is a simple syntax error typo problem. Perl can not find the function 'printt'. You need to fix all these.

Missing Modules

Modules are Perl's versions of libraries. There are modules available to do everything from read in CGI data to allowing you to write your script in Latin! To use them they need to be installed.

C:\>perl -c test.pl
Can't locate Some/Module.pm in @INC (@INC contains: 
C:/Perl/lib C:/Perl/site/lib .) at test.pl line 1.
BEGIN failed--compilation aborted at test.pl line 1.
C:\>

This is telling you that the module Some::Module required by a 'use Some::Module;" directive is not installed. You will probably need to install the module. To install a module see a detailed How-To for locally installing modules and installing modules on Win32. You *may* be able to get away with commenting out the 'use Some::Module;' directive to test compile you script but this will fail if you have code like 'print Some::Module->Cool_Function;' later on.

2. Are you using strict and warnings?

#!/usr/bin/perl -w # "-w" turns on all sorts of warnings about probable errors. 

use diagnostics;   # optional; causes warnings to be explained in greater detail.

use strict;        # generates compile and run-time errors for certain unsafe constructs.

While use strict and warnings are not required for any perl script to run (including CGI) they can help you sort out all manner of problems and are highly recommended. For a complete discussion of this see Use strict warnings and diagnostics or die

3. Are you using CGI.pm to parse your CGI input data?

The leading block of reusable code for CGI purposes is CGI.pm. In a nutshell there is a lot more to parsing CGI data than first meets the eye - both from the functionality and security point of view. If you can't give ten good reasons not to use CGI.pm then *use it*. It is part of the standard perl distribution so should be available on any system that has Perl. Using it to get your form parameters is as easy as:
use CGI;            # use CGI.pm
my $q = new CGI;    # create new CGI object

# now get value of a form parameter named 'name'
my $name     = $q->param('name');

# now get an array of values of all form parameters named 'option'
my @options  = $q->param('option');  

As you can see it is very easy to use CGI.pm. Good entry points to all things CGI.pm, and why you should be using it are: use CGI or die and no excuses for not using CGI.pm.

B. Uploading to the Server

1. Did you upload in ASCII mode?

Assuming all has gone to plan so far we are ready to upload the script to the server. The server is the computer that is actually connected to the WWW and whose job it is the serve up HTML documents and process CGI scripts. A Perl script is just a text file. Unfortunately different computer operating systems handle text files differently. On *nix the line ending is a line feed (\n). On Dos/Win it is a carriage return line feed (\r\n). On Macs it is a carriage return (\r). Clearly some conversion will be in order when sending files from one system type to another. Never fear the problem has been solved for you. All you need to do is to make the transfer (usually by FTP) in ASCII mode. In this mode the line endings will be automatically converted for you.

Where you can strike problems is when you transfer a file in *BINARY* mode. In this mode the file is sent exactly as is so line endings are not converted. This can and does lead to problems. Make sure you transfer in ASCII mode.

2. Did you upload to the correct directory?

Most servers have a specific directory used to store Perl CGI scripts. Typically this will be something like:

/www.mydomain.com/cgi-bin/

It makes sense that you will want to get your script into this directory. You may need to confer with your systems administrator about this. BTW I can personally recommend Cute FTP for Win32 as very easy to use for this task.

C. Setting Up on the Server

OK so now we have our script in the cgi-bin. We need to set it up so that it can be executed by the server when it is called. Here is how CGI works in a nutshell:

(i). Browser requests a resource via a URL that is aimed at a CGI script, say via a link like:

<a href="http://www.mydomain.com/cgi-bin/myscript.cgi">Run my script</a>

(ii). The server receives the request and goes "Ah ha this is a cgi script that I need to run!"

(iii). The server executes the script passing it data via the environment variables in the $ENV hash and via STDIN. The script can access this data and process it. The script generates output on STDOUT

(iv). The server takes the output of the CGI script from STDOUT and returns it to the browser pretending it received a request for a static HTML document.

We can have problems in any of these areas. Of course!

1. Is the HTML document that calls the script pointed at the correct place?

In order to execute the script you need to ask for it to be executed. If you link to the URL above and your cgi-bin happens to be called localcgi (like mine) then you will get a 404 Document not found error. Of course the server can not find the document in /www.mydomain.com/cgi-bin/, it is after all somewhere else!

2. Is the server configured to execute the script, and are the permissions set?

You can't execute an HTML document, well at least Perl can't compile it! You also don't want to return the code in your script to the browser when it is requested. So it is obvious that the server needs some way of deciding if it needs to return the requested document of execute it. As you might expect there are many ways to achieve this result. Your systems administrator should know the exact details for your system but here is a general guide.

Under *nix all files have permissions. They have permissions for the owner, the group, and everybody else. The permissions are 4=read, 2=write, 1=execute. Thus a permission of 7 means that the file can be read , written and executed as 4+2+1=7. A permission of 5 means the file can be read and executed as 4+1=5. As a general rule you will want permissions of 755 for your file which grants you read, write and execute permission and everyone else read and execute permission. You set this using the command chmod 755. Directories also have permissions which need to be correct but they should be fine if this is a standard cgi-bin.

Some servers are configured to recognise any file that ends in .pl, .pm, .cgi as a Perl executable (especially WIN32) and act accordingly. You will need to use one of these extensions if that is the case so the server knows to execute the file. Still other servers assume that all files in certain directories (ie cgi-bin) are executable. The bottom line is is all else fails then check with your sysadmin.

3. Is the shebang line pointed at the Perl interpreter?

Under *nix if a request for a  file  marked as executable is made then the first 2 bytes are examined for the #! shebang sequence. If this is found then the shell executes that file using the executable file it finds at the path following the #! Thus what this line:

#!/usr/bin/perl

 actually says is: "Dear shell, if this file is called and it has read and execute permission please execute it using the executable file in the /usr/bin/ dir called perl". If the Perl executable is not in the /usr/bin/ directory then the shell will complain about not being able to find Perl. Unless the script has permissions of at least 5 (read and execute) the shell/Perl will not be able to read it (as it must) in order to execute it (as we want)

A common Win32 -> *nix problem is \r chars from the Win32 line ending \r\n (CRLF) remaining. This is usually due to uploading in binary mode. Anyway a typical error is 'bad interpreter'. You get this message because the *nix shell is looking for /usr/bin/perl\r and the executable "perl\r" does not exist. The usual fix is:

perl -pi -e 's/\r//' script.pl

This does an implace edit that removes all \r chars. Alternatively you could do a symlink from perl to perl\r but if you were root and knew how to do that you wouldn't be reading this.

Some servers are configured not to allow the execution of CGI scripts for security reasons, once again talk to ye sysadmin.

D. Debugging

1. Before doing anything else add this code to the top of your script. 

If you add this code your script will run for long enough that any errors appear in the browser window making it easy to see the problem, presuming of course that you have followed the advice above! This will avoid you getting the less than useful 500 Internal Server Error  or Premature end of script headers/Malformed script headers messages. You add this code just below the shebang line and before everything else. That's everything else. The reason for this is to minimise the lines which can cause problems before we reliably direct output (including syntax errors) to the browser

#!/usr/bin/perl -wT

# ensure all fatals go to browser during debugging and set-up
# comment this BEGIN block out on production code for security
BEGIN {
    $|=1;
    print "Content-type: text/html\n\n";
    use CGI::Carp('fatalsToBrowser');
}

# all the rest of the code goes here

Because this code is in a BEGIN block it is executed before everything else. Even before most of the script is compiled. In it we do three vital things to ensure that all errors from this point on will appear in the browser window thus making debugging *much* easier. You could look in the server logs to get the same information but as you may or may not have access to them and they can be anywhere on the server it is easier to use this instead.

The $|=1; forces buffer flushing. This ensures that output from the script goes immediately to STDOUT. Let's just say trust me on this or read Suffering from Buffering for all the gory details.;

Next we print a complete valid header. In the HyperText Transport Protocol (the http bit in http://www.mydomain.com) every request and response needs a valid header. It may or may not include a body but it must have a header. The end of the header is recognised by the \n\n sequence. This prints one blank line. If, for any reason, a blank line is printed *before* the "Content-type: text/html\n\n" you will get a premature end of script header error. By printing a valid header we avoid this. Note that I have presumed you want to output HTML but you can also output GIFs, JPEGs etc using a different header that specifies these different data types. The headers our script generates without this block will appear in the top left of the browser window where we can check that they are as expected. If you don't see them and your script now works your problem is that you are not outputting any valid header info.

Finally we add CGI::Carp which essentially does the same as lines 1 and 2 but in my experience is not as reliable as desired. Combined with the two lines above virtually all errors appear in the browser window with an explanation.

While we are in this topic the -w switch on the shebang line provides heaps of free help. If you want more detail add the line use diagnostics; as shown:

#!/usr/bin/perl -wT

# comment this out on production code as it uses a lot
# of memory and you won't have any errors left anyway :-)
use diagnostics;
You may be wondering about the -T switch. Or you may not be. Regardless you want to use it. The -T switch switches on taint checking which is Perl's unique anti hacker mode which warns you if you are doing dangerous stuff that makes it easy to hack your server via your script. See The World Wide Web Security FAQ for more details.

2. Are you checking the return values from the functions built in to perl?

Most of the file and system functions set $! and have return values that you can test thus:
open(FILE, "</file.txt") or die "Error opening /file.txt Perl says: $!\n";
$! will contain an error message that will give you more information on where your program is going wrong. The perlfunc man page will give you more information on the return values from functions. It is very handy to know that the reason your script is not working is because you can not find/open/write to a file for instance.

3. Are the modules required by your script actually installed on the server?

As we saw earlier if you script uses modules they need to be installed. Some modules come with perl (like CGI.pm) but others need to be installed. If you have followed the advice so far you will get the typical:

Can't locate Some/Module.pm in @INC (@INC contains: 
C:/Perl/lib C:/Perl/site/lib .) at test.pl line 12.

error message in your browser window. Get you sysadmin to install them. Or see the installing modules links above.

4. Looking at the program as it runs.

Like any program a CGI may run but not produce the output you expect because of logic errors. Provided you have already printed a valid header you can add temporary debugging print statements within your code to print out variable values and flags to allow you to follow the progress of your code as it runs. My code often looks like this:

...
my $debug = 0;  # set to true for debugging
...
print "Variable \$my_var contains '$my_var'\n" if $debug;
...
sub foo {
    print "Running sub foo!\n" if $debug;
    ...
}

The $debug variable makes switching the debugging prints on and off easy - that way I can leave them in my code. The often come in handy again when you make changes.

Firing up the Debugger (*nix only)

For details on how to get a Perl CGI script to fire up a debugging window under X Windows on your local machine see Debugging a CGI by toma

5. Don't be afraid to Ask 

If you have tried all the stuff above and have an error you can't fix then Ask the Monks. Many Monks do this stuff for a living so can probably help. In fact you will often find that if you show you can help yourself people will fall over themselves trying to help you. Good luck and enjoy the journey.

Credits

The following monks have contributed to this document Albannach, sierrathedog04, converter

tachyon

s&&rsenoyhcatreve&&&s&n\w+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

Comment on CGI Help Guide
Download Code
Re: CGI Help Guide
by cjf (Parson) on May 09, 2002 at 08:31 UTC

    <shamelessnodeplug>Essential CGI Security Practices</shamelessnodeplug>

    Perhaps a bit more info on security would be in order as well? After all, the less MSA-quality scripts placed online, the better.

    Other than that, great tutorial++ :).

Re: CGI Help Guide
by ninja-joe (Monk) on May 12, 2002 at 03:21 UTC
    I think the 5th point, "Don't be afraid to Ask", is an excellent one. Technically, just about every Perl related question could be answered by diligently seeking out the answer in documentation, quickly in my case, but sometimes not so quickly. It just depends and how clear the documentation is on whatever you're having problems with. Documentation is mostly written by humans and its not perfect, and the people who write them are certainly not psychic - which would be especially bad for those of us who dance naked in front of our house pets (they're our pets for Christ's sake, it doesn't count, right?) ... but it's not like I do that or anything.

    My real point is that if you're starting CGI progamming, the more and more you try at solving your problems the easier it gets and question asking is a necessary part of the learning process, as well as exercising your manual reading skills.

    Another very helpful and semi-related node: How to RTFM
Interactive Command Line Form Parameter Debugging
by skyknight (Hermit) on Jul 25, 2003 at 17:46 UTC

    Oft times you'll invoke a CGI script, and while the script will in fact execute, it doesn't quite do what you were expecting. Ascertaining the nature of the problem can prove spectacularly frustrating when you have as many degrees of separation as you do in the scenario of invoking the script indirectly via a web browser. You type something into your browser, it sends a request to the server, the server invokes the script, takes back its output, and sends it back to the browser. There may even be additional intermediary steps that obscure the problem.

    Being able to cut both the web browser and web server out of the picture, thus isolating the script, can be quite useful. This will enable you to see the pure, raw output of your script, including the HTTP headers, as well as give you all of the output of your script, even in the event of a crash... far more useful than just seeing "INTERNAL SERVER ERROR" and only being able to ascertain the direct cause of the script crash, as opposed to seeing all output that preceded the crash. Of course, to get the full picture you need to be able to send in the same form parameters that you were previously doing through the request originating from the browser.

    To do this, you'll want to instantiate a CGI object with a file handle opened to a file that contains field/value pairs, or with STDIN and type in the values when you run the script.

    use CGI; open(FILE, "<data.txt") or die "gah!"; my $query = CGI->new(FILE); ...

    Where data.txt contains stuff like...

    foo=bar baz=biz

    This paradigm will aid in detection of the problem, potentially exposing previously hidden issues, as well as greatly expedite the run/debug/tweak cycle.

      I can also see this useful for doing Test::More harnesses for CGI scripts. Excellent pointer! ++!

      ------
      We are the carpenters and bricklayers of the Information Age.

      Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

      Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

Re: CGI Help Guide
by Anonymous Monk on Dec 17, 2004 at 03:49 UTC
    Sometimes you're sending in ASCII and you need to binmode STDOUT

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perltutorial [id://165253]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (6)
As of 2014-04-19 16:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (483 votes), past polls