Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

polishing up a json fetching script for weather data

by Aldebaran (Deacon)
on May 12, 2020 at 18:40 UTC ( #11116726=perlquestion: print w/replies, xml ) Need Help??

Aldebaran has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

I present with output, source, and questions:

This is a production script that I've been working on, and it's at a point where it can serve as an sscce for a pm thread. I've documented a lot of it with comments, but let me state that this attempts to get the weather details for a specific location as well as the relative position of the sun. I'll take any and all suggestions on how to smooth down the rough parts or ecstatic to replace it with something better. For example, I'd love to see what marto can do with mojo....

Let me roll out the log first. I unlink it and recreate it with every time the script runs. These data are only typical:

2020/05/11 14:23:44 INFO ./6.weather.pl 2020/05/11 14:23:45 INFO $VAR1 = [ { 'base' => { 'unitCode' => 'unit:m', 'value' => 910 }, 'amount' => 'FEW' }, { 'base' => { 'value' => 3050, 'unitCode' => 'unit:m' }, 'amount' => 'BKN' }, { 'base' => { 'unitCode' => 'unit:m', 'value' => 4880 }, 'amount' => 'BKN' }, { 'amount' => 'OVC', 'base' => { 'value' => 7620, 'unitCode' => 'unit:m' } } ]; 2020/05/11 14:23:45 INFO $VAR1 = undef; 2020/05/11 14:23:45 INFO $VAR1 = '2020-05-11T20:53:00+00:00'; 2020/05/11 14:23:45 INFO td is 2020-05-11 20:53:00 2020/05/11 14:23:45 INFO julian day is 2020-05-11 20:53:00 2020/05/11 14:23:46 INFO row sun is Sun 3h 16m 43s +18 9.7' 1.010 61. +305 24.656 Up 2020/05/11 14:23:46 INFO name: Sun 2020/05/11 14:23:46 INFO altitude: 61.305 2020/05/11 14:23:46 INFO azimuth: 24.656 2020/05/11 14:23:46 INFO visible?: Up

First of all, wow, right? We've got 3 different identified cloud layers. This makes me want to have this capability on my phone. Let's take one more look now that I've taken the time to write this up, so this is hot off the press:

2020/05/11 17:40:07 INFO ./6.weather.pl 2020/05/11 17:40:08 INFO $VAR1 = [ { 'base' => { 'unitCode' => 'unit:m', 'value' => 1220 }, 'amount' => 'FEW' }, { 'amount' => 'OVC', 'base' => { 'value' => 1680, 'unitCode' => 'unit:m' } } ]; 2020/05/11 17:40:08 INFO $VAR1 = undef; 2020/05/11 17:40:08 INFO $VAR1 = '2020-05-11T23:53:00+00:00'; 2020/05/11 17:40:08 INFO td is 2020-05-11 23:53:00 2020/05/11 17:40:08 INFO julian day is 2020-05-11 23:53:00 2020/05/11 17:40:09 INFO row sun is Sun 3h 17m 12s +18 11.6' 1.010 35 +.801 79.815 Up 2020/05/11 17:40:09 INFO name: Sun 2020/05/11 17:40:09 INFO altitude: 35.801 2020/05/11 17:40:09 INFO azimuth: 79.815 2020/05/11 17:40:09 INFO visible?: Up

It feels like the weather has changed since then. It feels like sunset to me, though there may be another ten degrees for the mathematical-idealized data I'm receiving. Here's the script:

#!/usr/bin/env perl use strict; use warnings; use DateTime::Format::Strptime; use REST::Client; use Data::Roundtrip qw/:all/; use 5.016; use Log::Log4perl; use Data::Dump; # get rid of old log my $file = '/home/hogan/Documents/hogan/logs/4.log4perl.txt'; unlink $file or warn "Could not unlink $file: $!"; my $log_conf4 = "/home/hogan/Documents/hogan/logs/conf_files/4.conf"; Log::Log4perl::init($log_conf4); #info my $logger = Log::Log4perl->get_logger(); $logger->info($0); # this is our fetcher, similar to LWP::UserAgent # but better suited for this kind of web service: REST my $rest = REST::Client->new() or die "failed to construct client"; # set the host $rest->setHost('https://api.weather.gov'); # note to self: KTTD is troutdale my $query = "stations/KPDX/observations/latest"; # make the request and check the response code, 200 is good my $response = $rest->GET($query) or die "failed to GET($query)"; if ( $rest->responseCode() != 200 ) { die "failed to GET(" . $rest->getHost() . "/$query) with " . $rest->responseCode(); } # we get back JSON my $jsonstr = $response->responseContent(); # convert JSON string to a perl variable my $pv = json2perl($jsonstr); if ( !defined $pv ) { die "something wrong with this alleged json : '$jsonstr'"; } # go to the interesting part my $var1 = $pv->{'properties'}->{'cloudLayers'}; $logger->info(perl2dump($var1)); ## failure #my %cl = %$var1; #my $cl1 = $cl{'amount'}; #say "cl 1 is $cl1"; # try heat index # try to pluck out the reference: my $var2 = $pv->{'properties'}->{'heatIndex'}; my %cl2 = %$var2; my $str2 = $cl2{'value'}; say "str 2 is $str2"; $logger->info(perl2dump($str2)); # this ain't right # let's get the timestamp my $var3 = $pv->{'properties'}->{'timestamp'}; say "var3 is $var3"; print perl2dump($var3); $logger->info(perl2dump($var3)); ## from bliako on same host (hope these are similar) # we have some dates in the data in ISO8601 format # this is a parser to convert that date to a DateTime # object which we can query about things (like seconds-unix-epoch) my $dateparser = DateTime::Format::Strptime->new( # parses 2020-04-26T06:00:00-05:00, # %F then literal T then %T then timezone offset pattern => '%FT%T%z' ) or die "failed to DateTime::Format::Strptime->new()"; my $time_dateobj = $dateparser->parse_datetime($var3) or die "parsing date failed"; my $td = $time_dateobj->strftime('%Y-%m-%d %H:%M:%S'); say "td is $td"; $logger->info("td is $td"); ## check to see what julian day it is use DateTime; my $julian=$time_dateobj->jd(); # this ain't right $logger->info("julian day is $td"); ### use retrieved time to figure out where sun is concurrently # use a perl automation tool that works for the $site* in question, # *which is written kinda funny use WWW::Mechanize; use HTML::TableExtract qw(tree); use open ':std', OUT => ':utf8'; my $site = 'http://www.fourmilab.ch/yoursky/cities.html'; my $mech = WWW::Mechanize->new; $mech->get($site); $mech->follow_link( text => 'Portland OR' ); $mech->set_fields(qw 'date 1'); $mech->set_fields( utc => $td ); $mech->click_button( value => "Update" ); my $te = 'HTML::TableExtract'->new; $te->parse( $mech->content ); my $table = ( $te->tables )[3]; my $table_tree = $table->tree; my $row = 2; # the sun my @sun = $table_tree->row($row)->as_text; $logger->info("row sun is @sun"); my $name = $table_tree->cell($row,0)->as_text; $logger->info("name: $name"); my $altitude = $table_tree->cell($row,4)->as_text; $logger->info("altitude: $altitude"); my $azimuth = $table_tree->cell($row,5)->as_text; $logger->info("azimuth: $azimuth"); my $visible = $table_tree->cell($row,6)->as_text; $logger->info("visible?: $visible"); $te->delete; __END__

Meanwhile, STDOUT is clogged like this in unscalable screenfuls:

Use of uninitialized value in hash element at /usr/share/perl5/HTML/El +ement.pm line 1674. Use of uninitialized value in hash element at /usr/share/perl5/HTML/El +ement.pm line 1674. Use of uninitialized value in hash element at /usr/share/perl5/HTML/El +ement.pm line 1674. Use of uninitialized value in hash element at /usr/share/perl5/HTML/El +ement.pm line 1674. Use of uninitialized value $ptag in string eq at /usr/share/perl5/HTML +/TreeBuilder.pm line 1071. Use of uninitialized value in hash element at /usr/share/perl5/HTML/El +ement.pm line 1674. Use of uninitialized value $ptag in string eq at /usr/share/perl5/HTML +/TreeBuilder.pm line 1071. $

Q1) How do I clear my terminal from all these warnings?

  • subquestion 1a) Can I use Log4Perl to intern these warnings in a file, or would it just crush my pretty log?
  • subquestion 1b) Can I use the command line to direct diagnostics to a file?
## failure #my %cl = %$var1; #my $cl1 = $cl{'amount'}; #say "cl 1 is $cl1";

Q2) How do I tease values out of this json structure more reliably than hunting and pecking?

Where am I going with this? Essentially in the direction that bliako suggested here, Re: using a stochastic matrix to simulate weather conditions:

You still need to figure out how to convert the received data to machine-readable format, re: "sunny", and, most importantly, figure out how to store received data for easy access over time (SQLite database?, csv files?, json files? perl variable files which you can later eval? - in order of my preference) - what search keys, what identifies a unique record? etc.

I can create whatever container I want to in the my_data folder that is defined by Path::Tiny in the script. Q3) Is it possible to create a human-readable json file with the cloud data, relevant values like temperature and windspeed, as well as the values retrieved by the WM part of the script?

Ultimately, I'd like to determine what conditions attain that will kill the exposed virus on fomites, with further considerations given for how they affect the mini-oven which is dashboard of my auto: fomites in a car.

I *still* have not seen data for how well this virus fares on surfaces as a function of temperature, moisture, exposure to wind, sunlight, although I have seen them in foreign language sources, particularly russian. What veracity these glimpses have is what I am exploring here.

Other than that, I've tried to come up with the julian date of the timestamp I get from the api, and that is not attaining yet...

w.r.t. the above:

Q1) How do I clear my terminal from all these warnings?

  • subquestion 1a) Can I use Log4Perl to intern these warnings in a file, or would it just crush my pretty log?
  • subquestion 1b) Can I use the command line to direct diagnostics to a file?

Q2) How do I tease values out of this json structure more reliably than hunting and pecking?

Q3) Is it possible to create a human-readable json file with the cloud data, relevant values like temperature and windspeed, as well as the values retrieved by the WM part of the script?

Q4) How do I convert the timestamp to julian days?

Q5) I'm gonna put in q5 as a placeholder, because I have feel like I'm missing something and will want to come back to with an update.

We always practice social distance in the monastery...I feel closer to the people here than the patriotic, unscientific, insouciant orcs I have to avoid in real life.

Thanks for your comments,

Replies are listed 'Best First'.
Re: polishing up a json fetching script for weather data
by marto (Cardinal) on May 12, 2020 at 20:51 UTC

    As requested, a fairly rough and ready proof of concept using Mojo:

    #!/usr/bin/perl use strict; use warnings; use feature 'say'; use Mojo::UserAgent; use DateTime::Format::Strptime; use open ':std', OUT => ':utf8'; my $url = 'https://api.weather.gov/stations/KPDX/observations/latest'; # the API docs says you must identify yourself, please make this somet +hing legit my $name = '(example.com, contact@example.com)'; my $ua = Mojo::UserAgent->new; $ua->transactor->name( $name ); # get JSON response my $json = $ua->get( $url )->res->json->{properties}; # for each cloud.... foreach my $cloud ( @{$json->{cloudLayers}} ){ say "$cloud->{amount}, $cloud->{base}{value}"; } # JD from JSON timestamp my $format = DateTime::Format::Strptime->new( pattern => '%FT%T%z'); my $dt = $format->parse_datetime( $json->{timestamp} ); say 'Julian Day: ' . $dt->jd; my $pturl = 'http://www.fourmilab.ch/cgi-bin/Yoursky?z=1&lat=45.5183& +ns=North&lon=122.676&ew=West'; # you wanted Julian date so it looks like date should be '2' from the +source. my $tx = $ua->post( $pturl => form => { utc => $dt->jd, date => '2' } +); my $sunrow = $tx->res->dom->at('center:nth-of-type(3) table tr:nth-of- +type(3)'); # output say 'Name:' . $sunrow->children->[0]->all_text; say 'Altitude: ' . $sunrow->children->[4]->text; say 'Azimuth: ' . $sunrow->children->[5]->text; say 'Visible: ' . $sunrow->children->[6]->text;

    Output:

    FEW, 760 SCT, 3350 BKN, 7620 Julian Day: 2458982.28680556 Name: Sun Altitude: 61.520 Azimuth: 21.289 Visible: Up

    There may be some typos in there, chalk that one up to the time of day here. Let me know if there are any problems.

    Update: for the 'Name' switch to all_text from text.

      As requested, a fairly rough and ready proof of concept using Mojo:

      Your script works perfectly, right out of the gate. I have tons of questions but don't want to muck up this thread by overreaching on this nice, neat result. I have been trying to imitate and extend it in other ways, but these results are slow to attain, but I don't want you to think I'm not trying:

      $ ls *mo* 1.1.mojo_api.pl 1.monk.tag.pl 2.3.mojo_json.pl 2.monk.tag.pl 1.1.mojo_json.pl 1.mo_pm.pl 2.4.mojo_json.pl 3.1.mojo_weathe +r.pl 1.2.mojo_api.pl 1.mo_pm.txt 2.5.mojo_json.pl 3.mojo_weather. +pl 1.mojo_json.pl 2.1.mojo_json.pl 2.mojo_json.pl 1.mojo_tree.pl 2.2.mojo_json.pl 2.mojo_weather.pl $

      Some of them didn't work for the strangest reasons. I had to get used to the strangeness in other cases. I will have run these hundreds of times of times before it dawned on me that the reason I never saw a value of rainfall other than undef is that no one on god's green earth has any idea what that value could be. (I wonder why it's there. A stubout for guesses?)

      I do want to pursue these ideas further, though, and have started another thread which generalizes this result: getting Sun info with Astro::Coord::ECI::Sun

      Thanks so much for lending your considerable expertise to the matter at hand....

        I don't understand the reason behind the various approaches rather than keep things clean and just sticking with the Mojo based solution for everything. I thought there would be a better, local way to get whatever data you are looking for, but I'm not an astronomer. If Astro::Coord::ECI::Sun does this then I don't see any problem in just calling that as many times as you require.

Re: polishing up a json fetching script for weather data
by marto (Cardinal) on May 12, 2020 at 19:22 UTC

    Can you post 4.conf so your example is runable? Without the logging I get:

    failed to GET(https://api.weather.gov/stations/KPDX/observations/lates +t) with 502 at ex.pl line 26.

    Do you have a URL which currently isn't failing?

      Can you post 4.conf so your example is runable?

      I use differing conf files based on whether I want DEBUG or INFO. These are modeled after the examples:

      ###################################################################### +######### # Log::Log4perl Conf + # ###################################################################### +######### log4perl.rootLogger = INFO, LOG1, SCREEN log4perl.appender.SCREEN = Log::Log4perl::Appender::Screen log4perl.appender.SCREEN.stderr = 0 log4perl.appender.SCREEN.layout = Log::Log4perl::Layout::PatternLayou +t log4perl.appender.SCREEN.layout.ConversionPattern = %m %n log4perl.appender.LOG1 = Log::Log4perl::Appender::File log4perl.appender.LOG1.filename = /home/hogan/Documents/hogan/logs/4. +log4perl.txt log4perl.appender.LOG1.mode = append log4perl.appender.LOG1.layout = Log::Log4perl::Layout::PatternLayou +t log4perl.appender.LOG1.layout.ConversionPattern = %d %p %m %n

      is 4.conf, and its partner, 3.conf only differs by a word:

      ###################################################################### +######### # Log::Log4perl Conf + # ###################################################################### +######### log4perl.rootLogger = DEBUG, LOG1, SCREEN log4perl.appender.SCREEN = Log::Log4perl::Appender::Screen log4perl.appender.SCREEN.stderr = 0 log4perl.appender.SCREEN.layout = Log::Log4perl::Layout::PatternLayou +t log4perl.appender.SCREEN.layout.ConversionPattern = %m %n log4perl.appender.LOG1 = Log::Log4perl::Appender::File log4perl.appender.LOG1.filename = /home/hogan/Documents/hogan/logs/3. +log4perl.txt log4perl.appender.LOG1.mode = append log4perl.appender.LOG1.layout = Log::Log4perl::Layout::PatternLayou +t log4perl.appender.LOG1.layout.ConversionPattern = %d %p %m %n
      Do you have a URL which currently isn't failing?

      Yes and no. the url was correct, but I had been using

      my $jsonstr = $ua->get( $query )->res->dom;

      to fetch the json, and it was unsuccessful for that reason. You use the correct url and fetching query in the newest script you posted.

Re: polishing up a json fetching script for weather data
by Fletch (Chancellor) on May 12, 2020 at 19:49 UTC

    WRT Q2, it's not perl but jq is really handy for pretty printing and manipulating JSON data from the command line. Basically you can think of it like awk but for files where the lines are JSON data rather than unstructured text. You can use curl to pull down a sample file, then interactively pick it apart with jq to figure out what bits you're interested in, and then automate whatever in perl.

    Additionally: For the warnings, the best way to get rid of them is to not trigger them. Check that you're actually getting values where you expect them before using things (alternately you could still be careless but mute them with no warnings 'undefined' in the smallest possible scope). That being said, they should be going to STDERR so if you use whatever your shell's syntax for that (e.g. myscript blah blah 2>errors).

    The cake is a lie.
    The cake is a lie.
    The cake is a lie.

      WRT Q2, it's not perl but jq is really handy for pretty printing and manipulating JSON data from the command line. Basically you can think of it like awk but for files where the lines are JSON data rather than unstructured text. You can use curl to pull down a sample file, then interactively pick it apart with jq to figure out what bits you're interested in, and then automate whatever in perl.

      Thanks for this tip, Fletch, it has really helped me pick apart these things. I folded them into system calls in a script where I built on what worked before:

      #!/usr/bin/perl use strict; use warnings; use open ':std', OUT => ':utf8'; #my $file = '1.weather.json'; #system ("cat $file | jq '[.properties]' | more"); #system ("cat $file | jq '[.properties.cloudLayers]' >2.txt"); #system ("cat $file | jq '[.properties.temperature]' >3.txt"); # these were effective commands ##different json, same host my $file = 'json_stuff/1.openapi.json'; #system ("cat $file | jq '[.]' >5.txt"); #system ("cat $file | jq '[.paths]' >6.txt"); system ("cat $file | jq '[.paths.get]' >7.txt"); __END__
      Additionally: For the warnings, the best way to get rid of them is to not trigger them. Check that you're actually getting values where you expect them before using things (alternately you could still be careless but mute them with no warnings 'undefined' in the smallest possible scope). That being said, they should be going to STDERR so if you use whatever your shell's syntax for that (e.g. myscript blah blah 2>errors).

      That's the ticket.

      $ ./6.weather.pl 2>errors
        system ("cat $file | jq '[.paths.get]' >7.txt");

        I very strongly recommend against doing it this way, for several reasons: First, you're using the shell's features, namely piping a file into jq when that's not necessary - jq filter files works just as well. Second, you're using the single-argument form of system, and interpolating a variable into that, opening yourself up to all sorts of possible issues; I describe that and several better alternatives here, in this case I might suggest capturex from IPC::System::Simple. Third, calling an external command in the first place - everything that jq can do, you can do just as well in Perl, by parsing the JSON file into a Perl data structure and then working with that data structure.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://11116726]
Approved by marto
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (7)
As of 2021-03-01 11:07 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?