Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

The Monastery Gates

( #131=superdoc: print w/replies, xml ) Need Help??

Donations gladly accepted

If you're new here please read PerlMonks FAQ
and Create a new user.

New Questions
Uncompress streaming gzip on the fly in LWP::UserAgent/WWW::Mechanize
1 direct reply — Read more / Contribute
by Your Mother
on Dec 18, 2018 at 17:15

    I include only the bare bones because I tried something like 20 different things without success and I'm embarrassed. :( Non-streaming requests are working perfectly with approximately this code. The endpoint for this code is a streaming quote and trade ticker. The URL shown just asks for the info for APPL. The compression—well, the ongoing, memory sane handling of it—is the only problem I have.

    The # $data is where the handling, or a core part of it, is missing.

    The closest I get streaming to work is gunzip from IO::Uncompress::Gunzip. It works fine but it's a standalone conversion which is goofy/impossible for an "infinite" stream of data; append, gunzip, append, gunzip… It must be uncompressed in toto. So a non-starter but I could see it works fine at least. Whatever the solution, I guess it will require IO handle truncation as it goes.

    I played with opening a pipe to gunzip(1), I tried every variation of Compress::Zlib I could think but wasn't able to get any data out or apparently even write/uncompress the data properly. I tried everything I could figure out with IO::Uncompress::Gunzip too. A few PM and SO nodes looked promising but I didn't find anything that seemed to apply directly or fix it. I even tried PerlIO::gzip. I'm just stick stupid on this. Maybe one of the attempts was close but I have no idea which, and, again, I'm embarrassed to share any of the *many*.

    The doc page https://www.ally.com/api/invest/documentation/streaming/. I tried to read the sample streaming code for other languages but there is no gzip handling surfaced in them. If they work, it's hidden in the libraries used.

    #!/usr/bin/env perl use strictures; no warnings "uninitialized"; use WWW::Mechanize; use WWW::OAuth; # You have to have a trading account to use the service- my $oauth = WWW::OAuth->new( client_id => "...", client_secret => "...", token => "...", token_secret => "..." ); my $mech = WWW::Mechanize->new( autocheck => undef ); $mech->add_handler( request_prepare => sub { $oauth->authenticate($_[0 +]) } ); $mech->default_header( Accept => "application/json" ); $mech->add_handler( response_data => sub { my ( $response, $ua, $h, $data ) = @_; # $data; # Handle the gzip data. $response->content(undef); 1; }); $mech->get("https://stream.tradeking.com/v1/market/quotes?symbols=AAPL +");

    On the plus side, the WWW::OAuth was trivial to mix in and works great for this.

    Thanks for looking!

Push array into array of arrays
5 direct replies — Read more / Contribute
by Anonymous Monk
on Dec 18, 2018 at 09:12
    push (@myArrofArray, \@myArray);

    1) I tried to print @myArrofArray, but i get this "ARRAY(0x7c8f60)" which I couldn't get any relevant results on Google. :(

    I tried to use perl debugger, x Dumper \@myArrofArray

    but there is still one original array in it, without the new (@myArray)

    May I know if the line is correct?

    If it is correct, then I have trace back to check maybe the array isn't being imported into array in the first place, as it was retrieved sql results.

    2) I am trying to get last array in the myArrofArray,

    push(@a,pop(@myArrofArray));

    print @a, I got ARRAY(0x858bd8)

    expected output: @a = ('x', 'y', 'z', 8,9,7);

    Can anyone point out the mistake?

    I can provide more information if it is needed. Thanks!!
LWP::UserAgent timeout
3 direct replies — Read more / Contribute
by IB2017
on Dec 17, 2018 at 08:04

    Hello

    I use the following to retrieve from server the version number of the latest available update of my software.

    my $ua = LWP::UserAgent->new; my $req = HTTP::Request->new(GET => $url); my $res = $ua->request($req); $ua->timeout(0.1);

    This works fine execpt if there is a connection problem. For example, the server has been down for the whole weekend now (!!!). In such a case, my GUI application - while running this peace of code - freezes for ca. 15 seconds. This is not acceptable. What I want is that if there is no response from the server, no matter the reasons, the application just skip this check. I tried to replace LWP::UserAgent with LWPx::ParanoidAgent, but this did not lead to any improvement. Any other idea?

Log::Any with die and carp
2 direct replies — Read more / Contribute
by learnedbyerror
on Dec 16, 2018 at 13:43

    Hello Oh Wise Monks

    The village idiot returns with another request for your guidance. I have long been a devotee of the most excellent Log::Log4perl. However, I am currently working on a module that I think may benefit others and will place it on CPAN. As such, I plan to change the log producer to Log::Any. For most of my logging calls, this is a trivial change. There are several cases where the change is not so trivial. They are replacing $logger->logdie (which I shouldn't be using in a module anyway ) and more importantly the Log4perl levels that handle Carp: $logger->log(carp|cluck|croak|confess).

    My question is, "what are your recommended means of integrating Log::Any where die and Carp functionalities are needed?"

    Some of the options are not too disruptive, replace $logger->logdie(...)with $logger->logcritical(...) && die(...)

    Whereas cluck or confess would be something more like

    $logger->critical(...); my $long_message = Carp::longmess; foreach my $line ( $long_message =~ m/^.+$/msg ) { $logger->critical( $line ); } confess(...)

    NOTE: untested code example, intended for representative use only

    While functional, it seems inelegant to have to add these boilerplate lines or add a subroutine or class method to handle this. Are their more elegant ways of handling this need with Log::Any?

    Thanks in advance for your sage advice!

    lbe

How to Extract Calendar Events in a Specified Date Range from 'Internet Calendaring' (.vcs, .ics) Files
2 direct replies — Read more / Contribute
by ozboomer
on Dec 15, 2018 at 03:38

    (..."Internet Calendaring"!? *Ugh* What a mouthful...)

    I have an Android smartphone and I use MyPhoneExplorer ("MPE") on a Windows 8 desktop to manipulate the calendar(s) in the phone.

    A feature of MPE is that it can export the calendar data in a few formats, being:-

    • vCalendar (.vcs);
    • iCal (.ics); and
    • XML (.xml)

    I have no knowledge of processing these formats... and so, I'm looking for some pointers on how I might approach dealing with the requirements I have to process this data.

    I'm hoping it makes sense to do something like:-

    Magic_Processing(input_file, events_structure) for date is startdate to enddate if exists events_structure{date} event_detail = events_structure{date} further_process(date, event_detail) endif endfor

    ...where Magic_Processing() is "aware" of the different types of events so the pseudocode I've shown actually works. This means the function would understand about:-

    • a 'one shot' event with a single occurrence date; OR
    • a repeating event, specified with: (a start date + no. of repeats) OR (a start date and an end date) OR ...other forms I don't (yet) know...

    ...or something like that.

    Simplified, I have a lot of repeating events... and I want to be able to specify a date range and have all the repeating events that will occur in that date range added to a list. Looking at the exported files, entries like:

    RRULE:FREQ=MONTHLY;BYDAY=2SA;INTERVAL=6
    (in an .ics file) are sort-of meaningless, straight off the bat.

    I understand RFC 5545 and possibly RFC 3283 are (probably?) the documents I should attack (at least, for the .ics files, which seem sort of the "most often seen"(!?))... but dealing with detailed RFCs like that is like trying to read microfilm through a mud-filled beer bottle to me... hence this request for some pointers on where to start.

    (Apologies if this is sacrilege to write in the Monastery) ...even if there's some Python code I could examine which would give some clues (as I'm trying to learn this language at present)...

    Thanks a heap for any thoughts or pointers.

Print text on the same line after match
4 direct replies — Read more / Contribute
by periodicalcoder
on Dec 14, 2018 at 15:16
    Hello all, it has been a while for me. You were all a huge help with my last project.

    This is actually a follow-up of that same project, and while I feel that this should be simple I cannot find (or don't know enough to adapt) a solution online. I have a text file with a line (there will only ever be a single line that matches) like this:

    N1*PE*COMPANY NAME INC*XX*123456~

    **edit: on Windows using Strawberry Perl with a .pl script** I need to extract the number at the end of this line, excluding the tilde, and add it to a variable so that $number = 123456. So far I have only been able to come up with the match regex, but I'm not sure how to get the text at the end. So far I have this which gives the entire line:

    perl -ne "print if /INC\*XX\*/" cr835.txt

    Inside my perl script I assume that I will use something like this, but of course including any recommended code changes:

    my $number = "perl -ne "print if /INC\*XX\*/" cr835.txt";

    My end goal is to detect this number and use it to rename the file. I already have the code tested for renaming the file with a timestamp and I just need help populating this variable.

    Thanks a bunch!

    peridicalcoder

regex: help for improvement
3 direct replies — Read more / Contribute
by frazap
on Dec 14, 2018 at 02:45
    I have names all composed of ascii characters, that I need to uniformize:
    • remove non letter (',.)
    • replace "-" by a space
    • start each word with an upper case
    • break camel case word: dosSantos -> Dos Santos

    I came with the code below that seems to work (as far as I can test it).

    My questions: how could I improved it ? (I think it will break if with unicode characters, what changes should I made to get it work with any character set ?)

    Thanks

    François

    use strict; use warnings; while ( my $t = <DATA> ) { chomp $t; printf "orig: %-30s translated: %s\n", $t, translate($t); } sub translate { my $str = shift; $str =~ tr/-/ /; #replace - with a space $str =~ tr/a-zA-Z/ /cs; #replace non letter with a space my @words = split( /\s+/, $str ); foreach my $w (@words) { #insert a space when a upper case is inside a word if ( $w =~ /\p{isLower}\p{isUpper}/ ) { my @all; while ( $w =~ m/\G(\p{isUpper}*\p{isLower}+)/g ) { push @all, $1; } $w = join( " ", @all ); } else { $w = ucfirst( lc($w) ); # we are using side effect of fore +ach loop } } return join( ' ', @words ); } __DATA__ Acierno James S., Jr. Acierno James, Jr. Ackermann-Hirschi L. Agatonovic-Jovini T. Alba-Castro Jose-Luis Alconada Verzini M. J. AlconadaVerzini M. J. Alvarez Fernandez A. Alvarez-Bolado Gonzalo Alvarez-Gonzalez B. AlvarezGonzalez B. AlvarezPiqueras D Amor Dos Santos S. P. Amor DosSantos S. P. AmorDosSantos S. P da Costa F. Barreiro Guimaraes Dano Hoffmann M. DanoHoffmann M. Dell' Acqua A. Dell' Asta L. Dell'Acqua A. Dell'Asta L. Dell'Omo Giacomo della Volp D. della Volpe D. Della Volpe D. DeRegie J. B. De Vivie Derendarz D. deRenstrom P. A. Bruckman Dupl'akova Nikoleta Duplakova Nikoleta Faucci Giannelli M. Fauccigiannelli M. FaucciGiannelli M. Yusuff I. Yusuff' I. Yao W-M Yao W-M. Yao W. -M Yao W. -M.
Mojolicious global variable
1 direct reply — Read more / Contribute
by bartrad
on Dec 13, 2018 at 14:53

    Evening monks, I'm after some of your wisdom.

    I have a Mojolicious lite app that works as I would expect for a single user. However, if two users are accessing it at the same time, the value of $router is changing depending on who submitted the request last.

    For example, if user A submits the form to query 'RouterA' then the information is displayed for that router. However, when user B then submits the form to query 'RouterB', the session for user A and user B displays the information for 'RouterB'. I figure it must be since I am declaring $router (and other things) globally but I'm not sure how to store this and pass it around the different routes I have...

    my ( $router, $cfg, $ip ); get '/' => sub { my $c = shift; $c->render( template => 'index' ); }; post '/' => sub { my $c = shift; ($router) = $c->param('router'); app->log->info("Router $router"); $cfg = get_config($router); $ip = $cfg->get_ip(); $c->render( template => 'result', router => $router, ip => $ip, ); } => 'index'; get '/info' => sub { my $c = shift; $c->render( template => 'result', router => $router, ip => $ip, ); }; get '/cards' => sub { my $c = shift; my %cards = $cfg->get_cards(); $c->render( template => 'cards', router => $router, cards => \%cards + ); }; app->start;

    The reason I have globally declared the variables is so I can pass them between the different routes and have the value of $router preserved and displayed on each resultant HTML page. Any help greatly appreciated as I'm scratching my head with this one!

Net::SSH::Perl and Net::SSH::Perl::Agent
2 direct replies — Read more / Contribute
by arc444
on Dec 13, 2018 at 14:16

    Hi. Can anyone advise how I can get my Net::SSH::Perl object to use a particular key I have identified in my Net::SSH::Perl::Agent object ?

    I have deliberatley simplified down - I will actually iterate over all available keys until I find the one I want ( based on 'comment' ) and then wish to use that particular key

    Im unable to try all keys as my target host only allows a few attempts, and I have many keys loaded into the agent ( reducing these available keys is not an option )

    Many Thanks !

    # Ensure the invoking process has a valid ssh-agent running, and is lo +aded with an appropriate key, or keys use warnings; use strict; use Net::SSH::Perl::Key; use Net::SSH::Perl::Agent; my $agent = Net::SSH::Perl::Agent->new(2); my ($key, $comment) = $agent->first_identity; use Net::SSH::Perl; my $ssh_user = 'myuser'; my $ssh_host = 'myhost'; ## How to use the above referenced $key in the following ssh connectio +n ?? ## my %ssh_params = ( debug => '1', options => [ "StrictHostKeyChecking n +o", "UserKnownHostsFile /dev/null" ] ); my $ssh = Net::SSH::Perl->new( $ssh_host, %ssh_params ); my $result = $ssh->login("$ssh_user"); if ( $result ) { print "Logged in as ${ssh_user} on host ${ssh_host}\n"; } else { print "Failed to login in as ${ssh_user} on host ${ssh_host}\n"; }
Perl launching process
6 direct replies — Read more / Contribute
by pwagyi
on Dec 13, 2018 at 05:17

    Greeting monks! I have been using perl system, backtick for some time to invoke other executables. But then I came across IPC::Cmd run, run_forked, I realize that it's somewhat better in a sense that I can easily access stdout, stderr and also exit code. So the question is when is it appropriate to use each system, backtick, or other modules? Is there best practice?

Killer byte tripping up Perl?
4 direct replies — Read more / Contribute
by echo5
on Dec 12, 2018 at 17:22

    I have a simple script that appears to show that some byte in the output is causing Perl to behave strangely.

    The script calls a command that coughs up 150 bytes or so of binary data. One stream of returned data behaves as expected. Another stream causes chaos.

    The script: $cmd = "/usr/local/bin/mycmd"; print "The cmd to be run is: $cmd \n"; open(CMD, "$cmd |" ) or die "Can't run '$cmd'\n$!\n"; while (<CMD>) { chomp; print "My raw output is: $_ \n"; $dataout = $_; print "My DATAOUT is: $dataout \n"; }

    When I run the above script in a "good" scenario I get the below output:

    The cmd to be run is: /usr/local/bin/mycmd My raw output is: &#65533; ` /11&#65533;%_$&#65533;%f$&#65533;&c$&#65 +533;%a$&#65533;%\$&c$&^$&#65533;&i$ My DATAOUT is: &#65533; ` /11&#65533;%_$&#65533;%f$&#65533;&c$&#65533; +%a$&#65533;%\$&c$&^$&#65533;&i$

    When I run the above script in a "bad" scenario I get the below output:

    The cmd to be run is: /usr/local/bin/mycmd My raw output is: My DATAOUT is: My raw output is: &#65533;0]&#65533;c &#65533;&#65533; My DATAOUT is: &#65533;0]&#65533;c &#65533;&#65533;

    Ultimately the goal is to ingest the data and process it using unpack but that was failing as $_ didn't contain data is should contain to process. The "bad" behavior above seems to show that there is some sort of "killer byte" being output from mycmd that throws a wrench into things. Below is the good and bad output in hex form via xxd. Is there a byte in there tripping up Perl?

    Good data stream: 0000000: 0202 00d0 0000 0000 0000 0000 0100 0000 ................ 0000010: 0100 0000 0100 0000 0100 0000 0100 0000 ................ 0000020: 0100 0000 0100 0000 0500 0000 0500 0000 ................ 0000030: 0500 0000 0500 0000 0500 0020 0100 0000 ........... .... 0000040: 0100 0000 0100 0000 0100 0000 0100 0000 ................ 0000050: 0100 0000 0100 0000 0500 0000 0500 0000 ................ 0000060: 0500 0000 0100 0000 0100 0000 0000 0000 ................ 0000070: 0200 0060 0100 0020 0000 0000 0100 2d00 ...`... ......-. 0000080: 0100 2f00 0100 3000 0000 0000 0100 0000 ../...0......... 0000090: 0000 0000 0102 ca25 0102 4a24 0102 bd25 .......%..J$...% 00000a0: 0102 5024 0102 da25 0102 4c24 0102 d525 ..P$...%..L$...% 00000b0: 0102 4c24 0102 c325 0102 4624 0102 e025 ..L$...%..F$...% 00000c0: 0102 4e24 0102 e225 0102 4824 0102 dd25 ..N$...%..H$...% 00000d0: 0102 5224 ..R$
    Bad data stream: 0000000: 020a 009c 0000 0000 0000 0000 0100 0000 ................ 0000010: 0100 0000 0100 0000 0100 0000 0100 0000 ................ 0000020: 0100 0000 0100 0000 0100 0000 0500 0000 ................ 0000030: 0500 0000 0500 0000 0500 0000 0500 0000 ................ 0000040: 0500 0000 0500 0000 0500 0000 0500 0000 ................ 0000050: 0500 0000 0500 0000 0500 0000 0500 0000 ................ 0000060: 0500 0000 0500 0000 0100 0000 0000 0000 ................ 0000070: 0100 3000 0000 0000 0100 005c 0100 00b4 ..0........\.... 0000080: 0000 0000 0100 0000 0000 0000 0200 0063 ...............c 0000090: 0100 0020 0000 0000 0103 9206 0103 8506 ... ............
Grep Pattern
4 direct replies — Read more / Contribute
by GotToBTru
on Dec 12, 2018 at 09:29

    I want to apply a F T T F repeating pattern as a filter to an array or list. Here is what I came up with:

    $i = 0; @result = grep { $i = 0 unless ($i<4); $i++%3 ? 1 : 0; } 0..12

    @result = 1,2,5,6,9,10

    It works but looks clunky. Any more elegant options?

    But God demonstrates His own love toward us, in that while we were yet sinners, Christ died for us. Romans 5:8 (NASB)

New Meditations
Inserting Code Before an -n/-p Loop
3 direct replies — Read more / Contribute
by haukex
on Dec 17, 2018 at 15:51

    Probably most people know about the "Eskimo greeting" "secret operator". I'm not sure if the following trick is common knowledge, but I just saw it for the first time in this blog post by Yary:

    $ perl -MO=Deparse -M'5;print "foo"' -ne '}{print "bar"' sub BEGIN { require 5; () } print 'foo'; LINE: while (defined($_ = readline ARGV)) { (); } { print 'bar'; }

    A neat little trick :-)

RFC: Set::Select: get intersection or union of sets; or more generally, the set of elements that are in one or more input sets
2 direct replies — Read more / Contribute
by kikuchiyo
on Dec 12, 2018 at 16:58

    If we have two sets, it is considered a solved problem to get their union or intersection or symmetric difference, there is even an entry in perlfaq4 about it. The situation is slightly more complicated if we have more than two input sets, because then it's a valid question to ask to e.g. get the set of elements that are in the first or second set but not in the third and fourth etc. The number of combinations grows rapidly with the number of input sets, and just writing ad hoc solutions to each little problem becomes infeasible. So a more general solution is needed - the hard part is designing the user interface so that it is able to express all the possible combinations of selections in a general and flexible, yet efficient and understandable manner. A cursory search of CPAN brought up several (abandoned?) modules in the Set::* namespace, but none of them was exactly what I needed.

    I have the outline of an attempted solution. It's an OO module that has a constructor to which the input sets can be fed, and one method called select, which accepts a selector string argument and emits (an arrayref of) the elements that match the selector. If we have 3 input sets, then the '110' selector string selects all elements that are in the first and second sets but not in the third.

    #!/usr/bin/perl { package Set::Select; use strict; use warnings; sub new { my ($class, $args, @sets) = @_; my $attr; $attr = $args->{key} if (ref $args eq 'HASH' and exists $args->{ke +y}); my $self; for my $i (0..$#sets) { for my $elem (@{$sets[$i]}) { my $key = defined $attr && ref $elem eq 'HASH' ? $elem->{$ +attr} : $elem; $self->{$key}->[1] //= $elem; $self->{$key}->[0] //= '0' x @sets; vec($self->{$key}->[0], $i, 8) = 0x31; } } bless $self, $class; } sub select { my ($self, $bits) = @_; return [map { $self->{$_}->[1] } grep { $self->{$_}->[0] =~ $bits +} keys %$self]; } } package main; use strict; use warnings; use Data::Dumper; my $x = Set::Select->new({}, [1, 3, 5, 7], [2, 3, 6, 7], [4, 5, 6, 7]) +; print Dumper $x->select($_) for qw/100 101 111 10. ... /; my $y = Set::Select->new({key => 'id' }, [{id => 1, value => 1}, {id => 3, value => 1}, {id => 5, value => +1}, {id => 7, value => 1}], [{id => 2, value => 2}, {id => 3, value => 2}, {id => 6, value => +2}, {id => 7, value => 2}], [{id => 4, value => 3}, {id => 5, value => 3}, {id => 6, value => +3}, {id => 7, value => 3}], ); print Dumper $y->select($_) for qw/100 101 111 10. ... /;

    A Venn diagram that may or may not make the intent clearer:

             .---.
            /  1  \
           |       |
        .--+--. .--+--.
       /   | 3 X 5 |   \
      |    |  / \  |    |
      |  2  \/ 7 \/  4  |
      |     |`---'|     |
      |      \ 6 /      |
       \      \ /      /
        `------^------'
    

    I think these selector strings as the primary (and only) user interface are better than the possible alternatives that come to mind: a verbose, ad hoc query language would have to be explained at length in the documention, tested carefully in the source, and parsed painfully at runtime, while a forest of arbitrarily named methods to select this or that subset would bloat the code needlessly and make it harder to use.

    Using regular expressions opens the door to abuse, but it also allows convenient and terse selector strings, makes the implementation efficient, and it's something people already know.

    If the elements are hashrefs (representing a record or object or something), there is a mode to use not the elements themselves but a named key inside them as the basis of selection, as the second example shows. This mode can be considered buggy as it is now, because only one version of a record with the same key is stored (in the example, some values are discarded. I don't have a good solution for this problem yet, partly because it would make the implementation slower and more complicated, partly because I don't know what would be the right thing to do.

    Questions:

    • Is this useful to anyone?
    • How to make it better?
    • What would be a good name if this were to become a module? I've tentatively chosen Set::Select but it may be too generic.
Log In?
Username:
Password:

What's my password?
Create A New User
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (2)
As of 2018-12-19 03:17 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How many stories does it take before you've heard them all?







    Results (83 votes). Check out past polls.

    Notices?