visualizing data in a table

[punkish]

Update1: I realize that PDL may be quite well suited for much of the above workflow, but I have had major issues with PDL's image plotting modules. Maybe I can do first part of the work with PDL, but am not very confident about generating images with it.

Update2: Added potential module names to tasks.

I want to visualize numbers in a table as a color ramp (or other color schemes). Below is how I would go about it. I am looking for suggestions for improving the process, and for modules that I can use.

Note: All numbers map to locations, hence, are values (for that particular variable) that can be placed on a rectangular image. In other words, the "rainfall" table has rainfall numbers that I want to convert to an image, one number per pixel.

1. $res = SELECT rainfall FROM rainfall_table WHERE <condition>. use DBI;
2. convert $res into an "interval" data set that can be mapped to a color table, otherwise their will be way too many colors, pointless in a web application anyway use Statistics::Descriptive; ??
3. choose a color scheme, preferably one that can follow good color principles, such as espoused by Cindy Brewer's http://colorbrewer.org. Or, at the least, be able to say -- "make me an image using a color ramp from 'dark blue (some rgb combo)' to 'bright pink (some rgb combo)'" use Imager::Color::Table; ??
4. generate the image use Imager or Image::Imlib2 or ImageMagick or GD; ??

Of course, the db work need DBI/DBD. But, what would I use to generate statistics on $res above? For example, converting the retrieved rows into interval groupings of different classes? Any suggestions for color table matching? And, what are the suggested modules for generating the image -- CPAN is full of image modules. I have ImageMagick already installed, but there is GD, the old stalwart, Imager, and a whole bunch more. Any toolbox with such tools already built in?

[BrowserUk]

For the color ramp part of the problem you might find ColorRamp1785 useful.

[punkish]

Gosh! ColorRamp1785 is beautiful. Many thanks. Update: Would be nice to have comments/annotation in that code. As is, it is not understandable by mortals such as myself. It does work beautifully.

Now, to fit the other parts of the chain together.

[BrowserUk]

Would be nice to have comments/annotation in that code.

BEGIN {
    my %map = (
         255 => sub{         0,             0,   $_[0] * 255 },
         510 => sub{         0,     $_[0]*255,           255 },
         765 => sub{         0,           255, (1-$_[0])*255 },
        1020 => sub{ $_[0]*255,           255,             0 },
        1275 => sub{       255, (1-$_[0])*255,             0 },
        1530 => sub{       255,             0,     $_[0]*255 },
        1785 => sub{       255,     $_[0]*255,           255 },
    );
    my @map = sort{ $::a <=> $::b } keys %map;
    sub colorRamp1785 {
        my( $v, $vmin, $vmax ) = @_;

        ## Peg $v to $vmax if it is greater than $vmax
        $v = $vmax if $v > $vmax;

        ## Or peg $v to $vmin if it is less tahn $vmin.
        $v = $vmin if $v < $vmin;

        ## Normalise $v relative to $vmax - $vmin
        $v = ( $v - $vmin ) / ( $vmax - $vmin );

        ## Scale it to the range 0 .. 1784
        $v *= 1785;

        ## And look up the appropriate rgb value
        ## And pack that into a 32-bit integer compatible with GD true
+color
        $v < $_ and return rgb2n( $map{ $_ }->( $v % 255 / 256 ) ) for
+ @map;
    }
}

The code provides a single function colorRamp1785() which takes 3 parameters:

1. $v: is the numeric value to map to a color on the color ramp.
2. $vmin is the minimum value $v will take. This will be mapped to the color black (rgb:0,0,0).
3. $vmax is the maximum value $v will take. This will be mapped to the color white (rgb:255,255,255)
• The sub pegs $v to $vmin or $vmax if it lies beyond their range.
• It then normalises $v in terms of the range between $vmin & $vmax.

  Ie. If you pass in ( 150, 100, 200 ), the $v becomes 0.5

• It then scales that to the 0 .. 1784 range of the color ramp

  Continuing the above example, $v now becomes 1785 * 0.5 = 892.5.

• Looks up the appropriate rgb value for that point on the color ramp

  Rather than having a huge 1785 entry lookup table, it uses a hash of subs to perform the mapping.

  • If the value is less than 256, then it represents the blue value directly with red and green both 0.

    $v = 0 => rgb(0,0,0); 1 => rgb(0,0,1); ... 255 => rgb(0,0,255)

    The color transitions from black to blue.

  • If the value is 256 .. 511, then (mod 256) it represents the green value directly with red = 0 and blue = 255.

    $v = 256 => rgb(0,0,255); 257 => rgb(0,1,255); ... 511 => rgb(0,255,255)

    The color transitions from blue to cyan.

  • If the value is 512 .. 783, then (mode 256) it inversely represents the blue value with red = 0 and green = 255.

    $v = 512 => rgb(0,255,255); 513 => rgb(0,255,254); ... 783 => rgb(0,255,0)

    The color transitions from cyan to green.

  • And so on through green to yellow to red to magenta to white.
• Packs the 3x8-bit rgb values to a 32-bit value compatible with truecolor images
• And returns it.

As a user, all you need to do is supply the numeric value + minimum and maximum and use the return to draw your plot.

Eg. If your minimum rainfall value is 0.5" and maximum 10", then to get the right color to plot the value 2.5",

my $plotColor = colorRamp1785( 2.5, 0.5, 10 );

If the drawing package you use needs discrete RGBs rather than packed, just remove the rgb2n() call from the return line.

Does that help?

[almut]

2. convert $res into an "interval" data set that can be mapped to a color table, otherwise their will be way too many colors

I think you just have to scale the values into the appropriate range, e.g. 0..15 for a ramp with 16 colors.

Here's an example where the sample values are in the range 0-1.  When your values ($res) are larger, just do something like 16 * $res / $max_value.

#!/usr/bin/perl
use strict;
use warnings;
use GD;

my @colorramp = ( # darkblue to yellow
    # R G B
    0x000000,
    0x000020,
    0x000040,
    0x000060,
    0x000080,
    0x0000a0,
    0x2000c0,
    0x4000e0,
    0x6000f0,
    0x8000d0,
    0xa00080,
    0xc00040,
    0xd04000,
    0xe08000,
    0xffc000,
    0xffff00,
);

my $x_size = 400;
my $y_size = 400;

my $img = GD::Image->new($x_size, $y_size, 1);

for my $x (0..$x_size-1) {
    for my $y (0..$y_size-1) {

        # some sample values in the range 0-1
        my $val = exp(-( (($x-$y)/2)**2 / 2000 + ($y-$x_size/2)**2 / 1
+0000 + rand()*0.1 ));

        my $idx = 16 * $val;   # scale appropriately

        $img->setPixel($x, $y, $colorramp[$idx]);
    }
}

open IMG, ">", "sample-image.png" or die $!;
binmode IMG;
print IMG $img->png();
close IMG;

[jethro]

Maybe I'm thinking too simple but I don't see any need for a statistics module. To visualize rainfall numbers you just have to decide the minimum and maximum and whether a linear or a logarithmic scale makes more sense

For the minimum you either take 0, some sensible smallest observable number or the smallest number you find in the table. If you want to be really clever, you might throw away the smallest 1% of the numbers to ignore extreme values (ok, that could be done by a statistics module)

Similarily either use the highest rainfall number in your table or some general maximum observable rainfall as maximum of your range

Caveat: using min/max numbers out of your table has the disadvantage that new data changes the intervals, visualized data-sets are not comparable anymore

Then you just need to decide whether you want a linear or logarithmic scale (maybe check out an almanac or other publication with rainfall tables for previous art)

Then just divide the range (or the logarithm of the range) by the number of colors and you have the interval (or the logarithm of the interval). Remember to cap numbers outside the range to the lowest or highest interval