Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Simultaneously reading from a file and serving data

by wazoox (Prior)
on Feb 27, 2014 at 17:29 UTC ( #1076424=perlquestion: print w/ replies, xml ) Need Help??
wazoox has asked for the wisdom of the Perl Monks concerning the following question:

Dear brethren,

I'm looking for an elegant way of reading data from a file on one hand, and serve it (preferably through a websocket) on the other hand. So far I managed to solve elegantly the problem of reading the output from several programs simultaneously by using on of the big powers of Unix : named pipes. However now I'm stuck on the next part: serving data. As reading from a pipe is blocking, and so is listening to a socket, I don't know how to get out of this.

So here's the code:

#!/usr/bin/perl use strict; use warnings; use Data::Dumper; use Net::WebSocket::Server; use JSON; my %data; my $json; # Net::WebSocket::Server->new( # listen => 8080, # on_connect => sub { # my ($serv, $conn) = @_; # $conn->on( # handshake => sub { # my ($conn, $handshake) = @_; # }, # utf8 => sub { # my ($conn) = @_; # $_->send_utf8($json) for $conn->server->connections; # }, # ); # }, # )->start; open my $fh, '<', 'toto1' or die "can't open fifo toto1: $?"; while ( my $line = <$fh> ) { my @list = ( split /\s+/, $line ); # skip empty lines next if $#list <= 1; shift @list if $list[0] eq ''; # test for iostat header next if $list[0] eq 'avg-cpu:'; next if $list[0] eq 'Device:'; # test for vmstat header next if $list[0] eq 'procs'; if ( $#list == 11 ) { ( undef, $data{io}{ $list[0] }{rrqms}, $data{io}{ $list[0] }{wrqms}, $data{io}{ $list[0] }{reads}, $data{io}{ $list[0] }{writes}, $data{io}{ $list[0] }{rsecs}, $data{io}{ $list[0] }{wsecs}, $data{io}{ $list[0] }{avgrqsz}, $data{io}{ $list[0] }{avgqusz}, $data{io}{ $list[0] }{await}, $data{io}{ $list[0] }{svctm}, $data{io}{ $list[0] }{util} ) = @list; } elsif ( $#list == 15 ) { ( $data{vm}{r}, $data{vm}{b}, $data{vm}{swpd}, $data{vm}{free}, $data{vm}{buff}, $data{vm}{cache}, $data{vm}{swapin}, $data{vm}{swapout}, $data{vm}{ioin}, $data{vm}{ioout}, $data{vm}{sysin}, $data{vm}{syscs}, $data{vm}{user}, $data{vm}{sys}, $data{vm}{idle}, $data{vm}{wait} ) = @list; } elsif ( $#list == 5 ) { $data{vm}{nice} = $list[1]; } $json = encode_json( \%data); sleep 1; }

Of course the script work by commenting out one of the two loops, either the websocket one, or the file reading one. To use this script, create a named pipe

mkfifo toto1

run continuously vmstat and/or iostat and redirect their output to the pipe:

vmstat 3 |tee toto1

and

iostat -x 3 | tee toto1.

I don't know how to get any further from there... maybe golang? :)

Comment on Simultaneously reading from a file and serving data
Select or Download Code
Re: Simultaneously reading from a file and serving data
by grondilu (Pilgrim) on Feb 27, 2014 at 17:45 UTC

    I'm no expert but: can't you just use a fork? I haven't used it for a while but IIRC it's actually not very hard to use and there are several nice examples in perlipc.

    Something like:

    if (my $pid = fork()) { # network stuff } else { # file stuff }
      The problem with a fork is that the %data won't be shared between processes...

        WARNING: I'm afraid this message is actually irrelevant. See EDIT note at the end.

        Well, you can have your two processes communicate with yet another named pipe. I really think you should (re-)read perlipc. There are lots of examples for this kind of stuff. Check out the open() command for instance. You can use it to fork and create a named pipe in the same time. See the "Safe Pipe Opens" section.

        Maybe something like this:

        if (my $pid = open(JSON, '|-') { # database stuff print JSON $json; } else { # this is the child fork. # STDIN here is actually JSON in the other fork. # network stuff my $json = <>; # some more network stuff I guess }

        EDIT I've just realized this wont solve your issue since the child process would block waiting for data from the parent process. It can't do that *AND* act as a network server at the same time. It only moves the blocking problem from one place to an other. So I'm sorry, I don't know.

Re: Simultaneously reading from a file and serving data
by kennethk (Monsignor) on Feb 27, 2014 at 19:11 UTC
    Couple of thoughts:
    • Split the tasks between two processes, and rather than sharing memory, use a shared database. You could even just have a shared file that contains your JSON and flock on updates.
    • Use threads and share the associated datastore (e.g. HoH) between the two using threads::shared
    • Use alarm to implement a timeout on your socket read.

    Any of these sound promising?


    #11929 First ask yourself `How would I do this without a computer?' Then have the computer do it the same way.

Re: Simultaneously reading from a file and serving data
by thargas (Deacon) on Feb 27, 2014 at 19:29 UTC
    Have you looked at IO::Select? It will allow you to have a loop which is "simultaneously" waiting for input from multiple sources and waiting for outputs to become ready for write and allowing for timeouts. Or you could use the underlying select call.

      select is definitely how I would do it, for several key reasons:

      1. It is architecturally uncomplicated:   the process is actually doing only one thing at a time, even though it is unpredictable what it might do next.
      2. You never have to worry about things being incomplete ... about accidentally sending someone a partial-message that you haven’t finished reading yet.
      3. You can easily prioritize what it is doing, according to the business requirement.
      4. Messy timing issues are generally avoided completely.

      The program itself is really a dispatcher, connected to everything else with nice, flexible pipes and sockets.   It goes to sleep, wakes up with a “honey-do list,” does its chores in whatever sequence it best sees fit, then goes back to sleep again.   As long as none of these things take a significant time to actually do, esp. relative to the others, it all works great.

Re: Simultaneously reading from a file and serving data
by davido (Archbishop) on Feb 27, 2014 at 19:29 UTC

    This isn't addressing your question, but one thing I noticed in your code is your verbose method of initializing %data. Note:

    use Test::More; use constant MY_KEYS => qw( rrqms wrqms reads writes rsecs wsecs avgrqsz avgqusz await svctm + util ); my $key = 'asdf'; my @list = ( 'a' .. 'l' ); my %data; ( undef, $data{io}{$key}{rrqms}, $data{io}{$key}{wrqms}, $data{io}{$key}{reads}, $data{io}{$key}{writes}, $data{io}{$key}{rsecs}, $data{io}{$key}{wsecs}, $data{io}{$key}{avgrqsz}, $data{io}{$key}{avgqusz}, $data{io}{$key}{await}, $data{io}{$key}{svctm}, $data{io}{$key}{util} ) = @list; my %data2; ( undef, @{ $data2{io}{$key} }{ +MY_KEYS } ) = @list; my %data3; @{ $data3{io}{$key} }{ +MY_KEYS } = @list[ 1 .. $#list ]; is_deeply \%data, \%data2, "Assigning an array to a slice."; is_deeply \%data, \%data3, "Assigning a slice to a slice."; done_testing(); __END__ __OUTPUT__ ok 1 - Assigning an array to a slice. ok 2 - Assigning a slice to a slice. 1..2

    Dave

Re: Simultaneously reading from a file and serving data
by karlgoethebier (Priest) on Feb 27, 2014 at 20:34 UTC

    BTW, in this context it might be interesting to show how i went over this bridge (AKA "Install Perl with threads" or so) using perlbrew:

    karls-mac-mini:~ karl$ perlbrew list perl-5.16.2 perl-5.16.3 perl-5.17.7 perl-5.18.0 perl-5.18.1 * perl-5.18.2 karls-mac-mini:~ karl$ perlbrew install --as perl-5.18.2threads -Duse +threads perl-5.18.2 Installing /Users/karl/perl5/perlbrew/build/perl-5.18.2 into ~/perl5/p +erlbrew/perls/perl-5.18.2threads This could take a while. You can run the following command on another +shell to track the status: tail -f ~/perl5/perlbrew/build.perl-5.18.2.log karls-mac-mini:~ karl$ perlbrew list perl-5.16.2 perl-5.16.3 perl-5.17.7 perl-5.18.0 perl-5.18.1 * perl-5.18.2 perl-5.18.2threads karls-mac-mini:~ karl$ perlbrew switch perl-5.18.2threads karls-mac-mini:~ karl$ perlbrew list perl-5.16.2 perl-5.16.3 perl-5.17.7 perl-5.18.0 perl-5.18.1 perl-5.18.2 * perl-5.18.2threads karls-mac-mini:~ karl$ perl -Mthreads ^C

    Seems to be pretty cool.

    Regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

Re: Simultaneously reading from a file and serving data
by BrowserUk (Pope) on Feb 27, 2014 at 21:41 UTC

    Completely untested, this should be close:

    #!/usr/bin/perl use strict; use warnings; use threads: + ## Added use threads::shared; + ## Added use Data::Dumper; use Net::WebSocket::Server; use JSON; my %data; my $json :shared; + ## Modded async { + ## Added Net::WebSocket::Server->new( listen => 8080, on_connect => sub { my ($serv, $conn) = @_; $conn->on( handshake => sub { my ($conn, $handshake) = @_; }, utf8 => sub { my ($conn) = @_; lock $json; + ## Added $_->send_utf8($json) for $conn->server->connection +s; }, ); }, )->start; }->detach; + ## Added open my $fh, '<', 'toto1' or die "can't open fifo toto1: $?"; while ( my $line = <$fh> ) { my @list = ( split /\s+/, $line ); # skip empty lines next if $#list <= 1; shift @list if $list[0] eq ''; # test for iostat header next if $list[0] eq 'avg-cpu:'; next if $list[0] eq 'Device:'; # test for vmstat header next if $list[0] eq 'procs'; if ( $#list == 11 ) { ( undef, $data{io}{ $list[0] }{rrqms}, $data{io}{ $list[0] }{wrqms}, $data{io}{ $list[0] }{reads}, $data{io}{ $list[0] }{writes}, $data{io}{ $list[0] }{rsecs}, $data{io}{ $list[0] }{wsecs}, $data{io}{ $list[0] }{avgrqsz}, $data{io}{ $list[0] }{avgqusz}, $data{io}{ $list[0] }{await}, $data{io}{ $list[0] }{svctm}, $data{io}{ $list[0] }{util} ) = @list; } elsif ( $#list == 15 ) { ( $data{vm}{r}, $data{vm}{b}, $data{vm}{swpd}, $data{vm}{free}, $data{vm}{buff}, $data{vm}{cache}, $data{vm}{swapin}, $data{vm}{swapout}, $data{vm}{ioin}, $data{vm}{ioout}, $data{vm}{sysin}, $data{vm}{syscs}, $data{vm}{user}, $data{vm}{sys}, $data{vm}{idle}, $data{vm}{wait} ) = @list; } elsif ( $#list == 5 ) { $data{vm}{nice} = $list[1]; } { ## + Added lock $json; ## + Added $json = encode_json( \%data); } ## + Added sleep 1; }

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      That mostly works :) I'm trying to get the client part working, and I'll share the lot, because a web-based system real-time system monitor may probably be of some use to others.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1076424]
Front-paged by toolic
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (10)
As of 2014-12-26 07:00 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (167 votes), past polls