Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Automated extraction of NOAA weather satellite images

by Scott7477 (Chaplain)
on May 29, 2007 at 22:25 UTC ( #618070=CUFP: print w/replies, xml ) Need Help??

NOAA updates its weather satellite images on the GOES website every 30 minutes, with images in infrared, visible light, and water vapor wavelengths; for both the eastern and western halves of the US. The script below grabs those images and saves them to a local file with a timestamp. Could prove useful for doing your own weather forecasting or whatever....
use strict; use LWP::Simple; use Time::Tiny; my $image; my $url; my %images = ( "eastconusir" => "http://www.goes.noaa.gov/GIFS/ECIR.JPG", "eastconusvis" => "http://www.goes.noaa.gov/GIFS/ECVS.JPG", "eastconuswv" => "http://www.goes.noaa.gov/GIFS/ECWV.JPG", "westconusir" => "http://www.goes.noaa.gov/GIFS/WCIR.JPG", "westconusvis" => "http://www.goes.noaa.gov/GIFS/WCVS.JPG", "westconusvwv" => "http://www.goes.noaa.gov/GIFS/WCWV.JPG", ); my $current_time = Time::Tiny->now; my $timestring = $current_time->hour.$current_time->minute.$curren +t_time->second; print $timestring."\n"; foreach my $key (keys %images) { print $key."\n"; print $images{$key}."\n"; my $status = getstore($images{$key},$key.$timestring.".jpg"); print $status."\n" ; };

Replies are listed 'Best First'.
Re: Automated extraction of NOAA weather satellite images
by graff (Chancellor) on May 30, 2007 at 02:40 UTC
    I had never heard of Time::Tiny before... If I understand its man page correctly, I think your code would create variable-length file names that might be hard to interpret reliably -- and might not be unique over the course of a day. That is, when you do:
    my $timestring = $current_time->hour.$current_time->minute.$current_ti +me->second;
    the time-stamp portion of each file name will range from "*.000.jpg" at midnight to "*.235959.jpg" at one second before midnight, with potential file-name collisions among sets of times like 01:23:45, 12:03:45 and 12:34:05 (which all would end up as "*.12345.jpg"), for example.

    Personally, I'd use the POSIX "strftime" function to set up fixed-width (zero-padded) strings for the file names -- and while I'm at it, might as well throw in the "ymd" date as well. Also, I think I'd rather use less repetition in the literal strings (looks like there was an inconsistency in your file names: an extra "v" in "westconusvwv").

    use strict; use LWP::Simple; use POSIX; my %image_url; for my $e_w ( qw/eastconus westconus/ ) { for my $typ ( qw/ir vs wv/ ) { $image_url{$e_w.$typ} = "http://www.goes.noaa.gov/GIFS/" . uc(substr($e_w,0,1)) . 'C' . $typ . '.JPG'; } } my $timestring = strftime( "%Y%m%d_%H%M%S", localtime ); print $timestring; for my $img ( keys %image_url ) { my $status = getstore( $image_url{$img}, join( '.', $img, $timestr +ing, 'jpg' )); print join( "\n", $img, $image_url{$img}, $status, "\n" ); }
    For that matter, it would be even simpler if I decided that the original four-letter NOAA file names were good enough for me...

    (updated the code to add a missing close-paren at line 16. Thanks, zentara!)

      Thanks for the feedback. I was looking at the POSIX module for including a timestamp in the saved filenames, but wanted to get a script up and running quickly and having never used the "strftime" function you mention Time::Tiny filled the bill. I agree with the potential problems you point out that the timestamps in my code have... I have updated my code to use the "strftime" function and the filenames now are very precise(for example "eastconusir20070530_074348.jpg"):
      #NOAA updates the images on the GOES website every 30 minutes use strict; use LWP::Simple; use POSIX; my $image; my $url; my %images = ( "eastconusir" => "http://www.goes.noaa.gov/GIFS/ECIR.JPG", "eastconusvis" => "http://www.goes.noaa.gov/GIFS/ECVS.JPG", "eastconuswv" => "http://www.goes.noaa.gov/GIFS/ECWV.JPG", "westconusir" => "http://www.goes.noaa.gov/GIFS/WCIR.JPG", "westconusvis" => "http://www.goes.noaa.gov/GIFS/WCVS.JPG", "westconusvwv" => "http://www.goes.noaa.gov/GIFS/WCWV.JPG", ); my $timestring = strftime( "%Y%m%d_%H%M%S", localtime ); print $timestring."\n"; foreach my $key (keys %images) { print $key."\n"; print $images{$key}."\n"; my $status = getstore($images{$key},$key.$timestring.".jpg"); print $status."\n" ; };


Re: Automated extraction of NOAA weather satellite images
by jasonk (Parson) on May 30, 2007 at 13:37 UTC

    As it happens, I wrote a very similar script a couple of weeks ago. Rather than using the current time though, mine builds a timestamp from the Last-Modified header of the response, so the timestamp is the last time the image was updated, and you can run it as often as you want without downloading duplicate images.

    #!/usr/local/bin/perl -w ################## # Copyright 2007 Jason Kohles (JSK), <email@jasonkohles.com> ################## use strict; use warnings; use LWP::Simple qw( getstore head ); use POSIX 'strftime'; #chdir( "/home/weather/noaa-goes-images" ) or die "Cannot chdir: $!"; for my $loc ( qw( EC WC ) ) { for my $type ( qw( VS IR WV ) ) { my $url = "http://www.goes.noaa.gov/GIFS/".$loc.$type.".JPG"; my $epoch = ( head( $url ) )[ 2 ] || next; my $time = strftime( '%F-%T', gmtime( $epoch ) ); my $file = lc( join( '-', $loc, $type, $time ).'.jpg' ); next if -f $file; print "Retrieving $url to $file\n"; getstore( $url, $file ); } }

    We're not surrounded, we're in a target-rich environment!
      It's a good idea to timestamp the images with Greenwich Mean Time since that's what NOAA uses: I updated my script with your modified timestamp.

      #NOAA updates the images on the GOES website every 30 minutes use strict; use LWP::Simple; use POSIX; my $image; my $url; my %images = ( "eastconusir" => "http://www.goes.noaa.gov/GIFS/ECIR.JPG", "eastconusvis" => "http://www.goes.noaa.gov/GIFS/ECVS.JPG", "eastconuswv" => "http://www.goes.noaa.gov/GIFS/ECWV.JPG", "westconusir" => "http://www.goes.noaa.gov/GIFS/WCIR.JPG", "westconusvis" => "http://www.goes.noaa.gov/GIFS/WCVS.JPG", "westconusvwv" => "http://www.goes.noaa.gov/GIFS/WCWV.JPG", ); foreach my $key (keys %images) { print $key."\n"; my $epoch = ( head( $images{$key} ) )[ 2 ] || next; my $timestring = strftime( "%Y%m%d_%H%M%S", gmtime( $epoch ) ); print $timestring."\n"; print $images{$key}."\n"; my $status = getstore($images{$key},$key.$timestring.".jpg"); print $status."\n" ; };

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: CUFP [id://618070]
Approved by graff
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (3)
As of 2021-06-14 22:18 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What does the "s" stand for in "perls"? (Whence perls)












    Results (66 votes). Check out past polls.

    Notices?