gnuplot and CGI

[kryberg]

Hi,

I want to use gnuplot to create graphs of a non-financial nature but very similar to the ones mentioned in this post - Re: Drawing Graphs. I've created many static graphs using gnuplot, Chart::Graph::Gnuplot, and GD::Graph. Now I want to generate graphs on the fly using parameters selected by the user.

How do I handle the output from the plotting process/what is the best method? Can I send the image data to the browser? Do I create temporary files? The ones referred to in the above referenced post appear to be temporary files. Do I write files to the directory and just href them?

I have the graphing process down, just not the output so any direction/suggestions would be appreciated.

[radiantmatrix]

One approach to this would be to code the graph-producing algorithm as a seperate CGI (for example, called graph.pl). Then, your core CGI script would produce an HTML page containing something like:

<img src="/cgi-bin/graph.pl?param=x&param2=y">

where your parameters contain the variable data you will be graphing. Done this way, your graph.pl can return e.g.:

Content-type: image/jpeg

###ACTUAL IMAGE DATA###

Doing this avoids the need for temp files, as the browser will take the output image data from your graph.pl and display it as an image.

[kryberg]

Thanks for the reply. I think your approach is exactly what I need to do.

I have not yet worked on the code to send the image data to the browser for this particular project. However, I've tried to do this in the past using GD, got it to work once, deleted that code when it turned out I didn't need it, tried it again and couldn't get it to work.

What I've tried in the past went something like:


print STDOUT $query->header(-type=>'image/gif');
binmode STDOUT;
print STDOUT $image->gif();

Is there a better approach?

[tmoertel]

Another option to consider is having your request handler generate the HTML and then "call back" to itself to generate the image:

    # generate either HTML or graph depending on CGI parms

    my $graph_parms = (new CGI)->param('graph');
    if ($graph_parms) { gen_graph($graph_parms) }
    else              { gen_html()              }

    sub gen_html {
        # ... generate other HTML ...
        # add IMG element that calls back to us to get image
        print qq(<img src="index.cgi?graph=..." ...>);
    }

    sub gen_graph {
        my $parms = shift;
        my $img_data = generate_graph( $parms );
        print "Content-Type: image/png\n",
              "Cache-Control: max-age=300\n\n",
              $img_data;
    }

The Cache-Control header may help reduce load on your server for frequently requested graphs.

Cheers,
Tom

[kryberg]

I've made progress but I'm still having trouble getting the image to display in the browser, I'm hoping I can get input from someone.

The following code prints the A OK message and then prints

Content-type: image/png Content-type: text/html 
Software error:
Failed to open pipe: No such file or directory

#!/usr/bin/perl -w

use CGI qw( :standard );
use CGI::Carp 'fatalsToBrowser';   

print "Content-type: text/plain\n\n";
print p( "A OK" );
print "Content-type: image/png\n\n";

my $data = "05056670.txt";

my $plot = plotdata();

sub plotdata {
  open my $graph => "| gnuplot" or die "Failed to open pipe: $!\n";
  my $graph;
  print $graph <<"gnu";
  set terminal png color
  set output
  set xdata time
  set timefmt "%Y%m%d"
  set key left top title "Legend" box
  set grid xtics ytics
  set yrange [700:]
  set format x "%Y"
  set xlabel "Year"
  set ylabel "Sodim, water, filtered, milligrams per liter"
  set title "05056670 Western Stump Lake Major Ions"
  plot "$data" using 2:3 title "P00930 Sodium dissolved"
gnu
  close $graph or die "Failed to close pipe: $!\n";
}

The following code appears to work at the command prompt because it prints the text and a bunch of strange characters, but it this is what the browser shows:

A OK 

Content-Type: image/png 

no image.


!/usr/bin/perl -w
use CGI qw( :standard );
use CGI::Carp 'fatalsToBrowser';  
my $q = new CGI;
print $q->header("text/html");
print p(" A OK ");
my $img = `gnuplot setcmds`;
print $q->header("image/png"), $img;

setcmds file:

set terminal png color
set output
set xdata time
set timefmt "%Y%m%d"
set key left top title "Legend" box
set grid xtics ytics
set yrange [700:]
set format x "%Y"
set xlabel "Year"
set ylabel "Sodim, water, filtered, milligrams per liter"
set title "05056670 Western Stump Lake Major Ions"
plot "05056670.txt" using 2:3 title "P00930 Sodium dissolved"

I'm sure I'm not doing something correctly, but I don't know what.

[tmoertel]

You can only send one response at a time, but you are trying to send back two responses smashed together (one blob of text and one image):

    print "Content-type: text/plain\n\n";
    print p( "A OK" );
    print "Content-type: image/png\n\n";

Only the first header is interpreted as a valid header, and hence the browser thinks it is seeing one big text response.

What you must do is send back one response per request. The trick is to set the src attribute of your first response's IMG element in such a way as to cause the client's browser to call your CGI back and ask for the image.

This is the idea:

1. The client's web browser requests your page.
2. Your CGI receives request and sends back a text/html response. The HTML includes an IMG element whose src attribute points back to your CGI and passes in an "I want the image" parameter.
3. The client's browser receives your response, parses the HTML, and then tries to load the image. In doing so, it calls your CGI again (the src attribute points to it), asking for the image.
4. Your CGI checks the request parameters and notices the "I want an image" parameter. The CGI then generates the image and sends back an image/png (or whatever type is appropriate) response.
5. The client receives the image and finishes rendering the page.
6. It's happy time. Dance.

For a complete example that uses this technique, see the script referenced at the end of The Mobile Weather Problem and its solution on my site.

Cheers,
Tom

[kryberg]

That sounds like a good possiblity - thanks

[talexb]

Your post brings back fond memories of what I was doing during the Internet boom five years ago.

But to answer your questions..

How do I handle the output from the plotting process/what is the best method?

I just re-directed the output from the gnuplot command to a temporary file with the .out suffix.

Can I send the image data to the browser?

Sure, but that's probably not what you want to do.

Do I create temporary files?

That's what I did. I just called a routine to cook up a temporary file name and used that in the gnuplot command file. The temporary files went into /var/tmp, and I even wrote a script to clean up that directory every an hour, having no idea that of course there was already a utility to do that (tmpwatch). At least I knew enough to use cron.

The ones referred to in the above referenced post appear to be temporary files. Do I write files to the directory and just href them?

Yup. Works great! Just be sure to clean up after yourself. Were I doing this again, I would use File::Temp to create temporary files. Back then I was a skilled C programmer fumbling my way around Perl, but learning fast.

Good luck!

[kryberg]

Thanks for your reply. I was excited to run across the link to your graphs because you used gnuplot and the web interface is the same idea as what I need to do.

It appears that I do need to send the image data to the browser though. This will be on a secure server that is out of my control. The people in charge have not been very supportive of on-the-fly graph generation. I've been doing some testing hoping I could find a way to write the file to the server but a cgi program cannot write a file because of the security restraints.

[kryberg]

Thanks to all the responses, I got this working having the image src call a separte cgi program.
<img src="test4.pl?sta=$station&const=$constituent" alt="$station $constituent" border="1">


#!/usr/bin/perl -w
# test4.pl for testing real-time graph capabilities
# November 9, 2004
use CGI qw( :standard );
use CGI::Carp 'fatalsToBrowser';
my $q = new CGI;
my $station = $q->param('sta');
my $constituent = $q->param('const');
my $combined = "$station" . "$constituent";
my $img = `/usr/local/bin/gnuplot $station $constituent $combined`;
print $q->header("image/png"), $img;

However, because I need to change the gnuplot set commands based on user input, I want to do this differently. Something like


#!/usr/bin/perl -w

# test5.pl for testing real-time graph capabilities
# November 17, 2004
use CGI qw( :standard );
use CGI::Carp 'fatalsToBrowser';    #will print error message to user
$CGI::POST_MAX = 512;   # limit maximum posting size to 512 bytes
$CGI::DISABLE_UPLOADS = 1; # disables uploading files entirely
my $q = new CGI;
my $data = "05056670.txt";
print $q->header("image/png\n\n");
my $img = plotdata();
sub plotdata {
  open my $graph => "| /usr/local/bin/gnuplot" or die "Failed to open 
+pipe: $!\n";
  print $graph <<"gnu";
  set terminal png color
  set output
  set xdata time
  set timefmt "%Y%m%d"
  set key left top title "Legend" box
  set grid xtics ytics
  set yrange [700:]
  set format x "%Y"
  set xlabel "Year"
  set ylabel "Sodium, water, filtered, milligrams per liter"
  set title "05056670 Western Stump Lake Major Ions"
  plot "$data" using 2:3 title "P00930 Sodium dissolved"
gnu
  close $graph or die "Failed to close pipe: $!\n";
}


When I try this, I get:
�PNG

IHDR�
�����sBIT�Lˢ0PLTE�����U�U��UUU��U�UUU�UUU������������I�N�.tEXtSoftwaregnuplot Unix version 3.7 patchlevel 1C����IDATx��Kң���`�3��&5��e�'5�e�HJ@ �!����tt�� �?���i � � � ���C���ۗn�4��o�a���m��]�����$붛l�ϐ>�a~�o�f&����w0߇�N�����ܺ ����aٍۇ):M��}ݍ:��MQ�u���aJn�>u�F}z�YO/�$����$Gg�\5l��=o����Oe����-L�f�e�Ƨ�’�.��&�<�)��9�դ�1���mt6�v0Qf
N���.T��#F�}�!���YuS�ԏzc���n�@7�(�M�r;�

in the browser.

What am I doing wrong?