Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Real time data graphs.

by enhering (Sexton)
on Oct 07, 2006 at 17:28 UTC ( #576881=perlquestion: print w/ replies, xml ) Need Help??
enhering has asked for the wisdom of the Perl Monks concerning the following question:

Hi. I'm building a data aquisition system in Perl that colects data at the rate of 10 cicles per second, and I'd like to show that in a graph on a Tk window.

If I use Tk::Canvas and create boxes or dots for each aquired data point, soon I'll fill all the memory because, if I understood correctly, each of those points is an entity on Tk.

Using Imager with Tk::photo was tried too, but the convertion from Imager format to PNG, as stated on Tk::Photo example, using base64 takes too long.

GD was also tried, but we need more functionality for the generated graphics.

In fact, what I'm looking for is a suggestion of a way to draw primitives on a canvas that are not items or objects, as was done in the past on VGA screens. The Tk::Photo + Imager solution would be fine if the conversion did not take so long.

Can you suggest me a way of solving this problem?


The following piece of code examples the problem. In a athlon64 notebook it takes 70% of the CPU to run.
#!/usr/bin/perl use strict; use Tk; use Tk::Photo; use MIME::Base64; use Tk::PNG; use Imager; my $xsize = 200; my $ysize = 200; my $cicle; my ($x, $y) = (int($xsize/2), int($ysize/2)); my $image = Imager->new(xsize=>$xsize, ysize=>$ysize); my $blue = Imager::Color->new( 255, 255, 255 ); my $image_data; $image->write(data =>\$image_data, type=>'png') or die "Cannot save image: ", $image->errstr; $image_data = encode_base64($image_data); my $main = MainWindow->new; my $tk_image = $main->Photo(-data => $image_data); my $screen = $main->Label(-image=>$tk_image)->pack; my $id = $main->repeat(100,\&move); MainLoop; sub move { my ($dx, $dy); my $i = int(rand(100)); if (($i/5) == int($i/5)) { $dx = 0; } elsif (($i/2) == int($i/2)) { $dx = 1; } else { $dx = -1; } my $j = int(rand(100)); if (($j/5) == int($j/5)) { $dy = 0; } elsif (($j/2) == int($j/2)) { $dy = 1; } else { $dy = -1; } if (($x + $dx) > $xsize) { $x = $xsize; } elsif (($x + $dx) < 0) { $x = 0; } else { $x += $dx; } if (($y + $dy) > $ysize) { $y = $ysize; } elsif (($y + $dy) < 0) { $y = 0; } else { $y += $dy; } $image->setpixel(x=>$x, y=>$y, color=>$blue); $cicle++; print "Cicle $cicle. Coords: $x, $y. Diference $dx, $dy.\n"; $image->write(data =>\$image_data, type=>'png') or die "Cannot save image: ", $image->errstr; $image_data = encode_base64($image_data); $tk_image->configure(-data => $image_data); $screen->configure(-image=>$tk_image); }

Comment on Real time data graphs.
Download Code
Re: Real time data graphs.
by Joost (Canon) on Oct 07, 2006 at 18:02 UTC
Re: Real time data graphs.
by RobPayne (Chaplain) on Oct 07, 2006 at 19:06 UTC
    enhering, have you tested using Tk::Canvas? How much memory did it take up to graph the number of points that you expect to plot? It doesn't make logical sense that you will necessarily fill all available memory because each acquired data point takes some amount of memory, because at some point you are going to hit "the end" of your graphing period.

    The length of your graphing period limits the amount of memory that you will need. (Points that are no longer graphed should no longer take up memory for Tk::Canvas.) That might mean that the memory requirements are very large, if your graphing period is long, but it will be finite. Whether your very long graphing period (assumption) fills all of memory is a question of how much memory is required to graph all of the data points that you have at that length.

    That's why I asked what you had measured in terms of memory requirements for any testing that you had already done.

      RobPayne, I may need to monitor the temperature of a system for days, at 864000 points per day. The only possibility I can see of using Tk::Canvas in this situation is by working with as many points as there are horizontal pixels in the canvas and reposition them when the graph fills up.
        You might want to look at rrdtool (RRD::Simple) and it's ability to summarize historical data. The issue may be how Tk::Canvas uses memory per point, but the limitation should be screen size before it is memory. If you plot ten points per second, at a screen resolution of 1600x1200, you can only print two days worth of data if you plot every point on the screen before you run out of space.

        You might want to take a look at the way mrtg graphs historical data, it limits database and graph size by graphing less granular data the older the further the data is in the past.

Re: Real time data graphs.
by BrowserUk (Pope) on Oct 07, 2006 at 20:49 UTC
    If I use Tk::Canvas and create boxes or dots for each aquired data point, soon I'll fill all the memory

    How about using a line item, and use the itemconfigure method to add new points to the line? That way you would only use one entity and get a line graph of the temp against time.

    GD was also tried, but we need more functionality for the generated graphics.

    What beside plotting points did you need for your graph that GD didn't provide?

    If GD really cannot provide what you need for legends, axis and stuff (though I cannot see why it wouldn't), the you might consider positioning the Tk::Photo with just the points plotted over the top of a tk::Canvas on which you draws the axis, legends etc.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Real time data graphs.
by Khen1950fx (Canon) on Oct 07, 2006 at 21:12 UTC
    If you like Tk::Photo but just want to speed it up, then how about trying Tk::Task. It'll break up the data into smaller pieces to "allow multiple tasks to proceed at once"and handle events between pieces. It would be more responsive. See: Tk::Task
Re: Real time data graphs.
by fmerges (Chaplain) on Oct 07, 2006 at 21:39 UTC

    Hi,

    I'm using rrdtool, for meteorological data an also for general telemetry data, as for example, hidrostatic system, telescope tracking, and a large of etceteras...

    Take also a look at PGPLOT and PGPLOT::Simple if you need to do some special temperatures related graphs...

    Regards,

    fmerges at irc.freenode.net
Re: Real time data graphs.
by Koosemose (Pilgrim) on Oct 08, 2006 at 03:31 UTC

    Just an idea for an alternate way to go about it. Consider having Tk::Canvas only show a finite time period, say the last hour for example, Continue to save the data points in an array, and at program completion use Image::Magick or some similar graphics generating package generate a graph containing the full graph.

    Not certain if it would work for your particular task, just figured I'd offer another potential way to attack the problem, and it might help solve the problem RobPayne mentioned of only a limited amount of space to draw in.

    Just Another Perl Alchemist
      The idea is to show only the last n minute period of aquisition and n should be configurable by the user. With the resources available now, the best way to do this seems to be by working with as many Tk points as there are horizontal pixels in the graph and make a transformation on the data so that the data points are represented by Tk points.

      But this is makes things unnecessarily complex. The best would be to be able to use imager primitives directly on a canvas, in some way.

        While poking around in the Tk Docs, I noticed Tk::Photo can take a -data option, I was attempting to see if this could lead to a solution by using Imagemagick to generate an image in memory, and use it as the input for the -data option, and just update it via a configure, unfortunately I can't figure out how to get Imagemagick to output the image data in a compatible format (it's ImagetoBlob method doesn't seem to do it quite right) I'm not familiar with any image packages other than Imagemagick, but if you are this may be a valuable route to explore, as you will only have a single image to update.

        Also tried to do a simple writing an image with Imagemagick, as it seems fairly quick at it, and can draw primitives directly as you want. Eh, I'm a little to tired to try offering advice, can't even get my test script to function properly. If you're still having issues by tomorow I'll see if I can't beat Tk and Imagemagick into submission :)

        Happy Coding

        Just Another Perl Alchemist
Re: Real time data graphs.
by zentara (Archbishop) on Oct 08, 2006 at 11:23 UTC
    I agree with BrowserUK, use a line( use the smooth option), it should be fast to update. Additionally, break your day into smaller periods, say hours, and seamlessly display them side-by-side on the canvas. That way, each update would only need to rebuild that hour's line segments. An hour of seconds would be 3600 segments. If that is too slow, break it into tenth's of an hour.

    See ztk-roller-coaster-simulation for a demo of how to make the segments.


    I'm not really a human, but I play one on earth. Cogito ergo sum a bum
Re: Real time data graphs.
by vkon (Deacon) on Oct 08, 2006 at 17:38 UTC
    BLT package is graph expert for Tk, see http://vkonovalov.ru/cgi-bin/perl-tcltk-wiki.cgi/38.

    This requires Tcl/Tk (via Tcl::Tk CPAN module), so you will have much better Tk library only if you have resources to move further, so this is not always an option for perl/Tk people who afraid to upgrade.

Re: Real time data graphs.
by dwhite20899 (Friar) on Oct 08, 2006 at 21:19 UTC
    How often do you actually need to view the real time data?

    When collected data every 32 ms for a project, I had a thread for collection that kept that cycle, and a thread for visualization that refreshed every 1, 2, or 5 seconds depending on how often a human needed to see it.

    Can you afford to show the data points at a lower rate than you collect it? Or graph every 10th point?

Re: Real time data graphs.
by tonyc (Pilgrim) on Oct 08, 2006 at 23:34 UTC

    I've looked at adding a faster mechanism to get an Imager image into a Tk::Photo, but Tk's internals scared me. Maybe it's time to look again.

    I'd planned on adding an Imager wrapper (at the C level) around a Tk::Photo object which would allow you to use Imager methods to draw directly to a Tk::Photo.

    I've done something similar with the unreleased Imager::SDL (or viewcvs).

      That would be great!
Re: Real time data graphs.
by zentara (Archbishop) on Oct 09, 2006 at 14:30 UTC
    See Tk Realtime data aquisition for a demo. It's just the basics, but it shows how to use the coords method to update a curve, and breaking it into segments. It would be nice to add a "floating y-axis" so as you scrolled, you could still read the y-axis value.

    I'm not really a human, but I play one on earth. Cogito ergo sum a bum

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://576881]
Approved by Corion
Front-paged by bart
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (7)
As of 2014-11-29 00:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (200 votes), past polls