Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

WebSocket idle

by chris212 (Scribe)
on Jan 03, 2018 at 01:29 UTC ( #1206571=perlquestion: print w/replies, xml ) Need Help??
chris212 has asked for the wisdom of the Perl Monks concerning the following question:

I'm creating a script that uses a websocket. It will constantly process many frames all the time. My script will collect all the data from those frames, and periodically perform some calculations on that data it maintains. I had it working so my calculations would run every second, and my script seemed to keep up with the data collection ok. However, that doesn't seem like the best method.

I think ideally, my calculations would run every time all frames are collected and the websocket is idle, even if just for a few milliseconds. That way I know my calculations are using the most current data received, and my 1 second timer won't be blocking the websocket from reading the frames it has already received. Then I would also know my calculations are run as frequently as possible rather than a fixed 1 second timer.

I can't seem to find a websocket module that has an idle event. It doesn't even need to use events. Even something with a function that will process all received frames when called and return when completed should work fine. Then I could just put that in a loop with my calculations. I can't seem to find anything like that though. Thanks monks in advance for any help.

Replies are listed 'Best First'.
Re: WebSocket idle
by haukex (Abbot) on Jan 03, 2018 at 19:29 UTC

    I'm not sure because you haven't shown any code, but it sounds to me like you might be using a blocking sleep call or something like it? Just a thought: have you considered using non-blocking I/O in an event loop? I've successfully used POE several times now, it has a little bit of a learning curve but if you get into it, it's pretty useful. Its Cookbook even includes an example Websocket client.

      I've been using event loops, but I haven't been able to get the loop to triage to my sub when there are no other events. I've only been able to force my sub to run as a periodic event. Here is the code I have now, but it doesn't actually read any frames. Maybe the subscription isn't being sent correctly? It shows the logic I am trying to accomplish, though.

      #!/usr/bin/perl use strict; use IO::Socket::SSL; use IO::Framed; use Net::WebSocket::Frame::text; use Net::WebSocket::Parser; use Net::WebSocket::Endpoint::Client; use Net::WebSocket::Handshake::Client (); use Net::WebSocket::HTTP_R (); use HTTP::Response; use JSON::XS; use Data::Dumper; my @prods = qw(BTC-USD LTC-USD ETH-USD); my $host = 'ws-feed.gdax.com'; my $wssurl = "wss://$host/"; my $port = 443; my $inet = IO::Socket::SSL->new("$host:$port") or die "failed connect or ssl handshake: $!\n$SSL_ERROR"; $inet->blocking(0); my $handshake = Net::WebSocket::Handshake::Client->new( uri => $wssurl ); print "REQ HEADERS:\n".$handshake->to_string()."\n"; syswrite $inet, $handshake->to_string() or die $!; my $buffer = ''; my $cnt = 0; my $line = <$inet>; while($line ne "\r\n") { $buffer.= $line; $line = <$inet>; } print "RESP HEADERS:\n$buffer\n"; my $resp = HTTP::Response->parse($buffer); Net::WebSocket::HTTP_R::handshake_consume_response( $handshake, $resp +); #See below about IO::Framed my $iof_r = IO::Framed->new($inet); $iof_r->allow_empty_read(); my $parser = Net::WebSocket::Parser->new( $iof_r ); my $iof_w = IO::Framed->new($inet); my $ept = Net::WebSocket::Endpoint::Client->new( parser => $parser, out => $iof_w, ); my $subscribe = { type => 'subscribe', product_ids => \@prods, channels => ['full'] }; my $payload = encode_json($subscribe); print $payload."\n"; $iof_w->write( Net::WebSocket::Frame::text->new( payload => $payload ) ); while(1) { while(1) { # we are catching up print "get next frame\n"; my $frame = $ept->get_next_message(); print "\$frame = \"$frame\"\n"; last unless($frame); load_frame(decode_json($frame)); } # we are caught up somethinghard(); # catch up again } sub somethinghard { print "something hard\n"; sleep 1; } sub load_frame { my ($frame) = @_; my $data = decode_json($frame); print Dumper($data); }

        What you've shown isn't really the kind of event loop I meant (see e.g. POE) - but that might not be necessary in this case. My understanding from the somewhat sparse documentation (the module is apparently still in beta) and a look at some of the code is that it will return undef if there is currently no message available. Without having the time to test myself at the moment, it sounds to me like this is the "idle" event you are looking for. I don't think you need nested loops either, a single loop with an if/elsif after get_next_message should be enough. Note there is also Time::HiRes that you can use to keep precise track of the timings if you like. And be careful with sleeps, depending on the message frequency, you may not need them at all and you might unnecessarily block your code. And a final nitpick: you seem to be using the term "frame" when you mean "message", I recommend sorting that out so there is no confusion when reading the code.

Re: WebSocket idle
by stevieb (Abbot) on Jan 03, 2018 at 01:59 UTC

    You've been here a little while and have a decent amount of posts.

    Please re-read the post you've done here, and reformat it with <p></p> tags so that it's readable.

    As is, it's a wall of words that the vast majority won't bother reading (myself included).

      It felt like one paragraph when I was writing it, but hopefully that spacing makes it less scary.
Re: WebSocket idle
by chris212 (Scribe) on Jan 20, 2018 at 03:00 UTC

    Here are some changes I needed, I think for SSL renegotiation. I wasn't able to find much about non-blocking sockets with SSL, so hopefully someone can confirm if I did this right.

    #!/usr/bin/perl use strict; use warnings; use IO::Socket::SSL; use IO::Framed; use IO::Select; use Net::WebSocket::Frame::text; use Net::WebSocket::Parser; use Net::WebSocket::Endpoint::Client; use Net::WebSocket::Handshake::Client (); use Net::WebSocket::HTTP_R (); use HTTP::Response; use JSON::XS; use Data::Dumper; use Time::HiRes qw(time usleep); my @prods = qw(BTC-USD LTC-USD ETH-USD); my $host = 'ws-feed.gdax.com'; my $wssurl = "wss://$host/"; my $port = 443; my $inet = IO::Socket::SSL->new("$host:$port") or die "failed connect or ssl handshake: $!\n$SSL_ERROR"; my $handshake = Net::WebSocket::Handshake::Client->new( uri => $wssurl ); print "REQ HEADERS:\n".$handshake->to_string()."\n"; syswrite $inet, $handshake->to_string() or die $!; my $buffer = ''; my $cnt = 0; my $line = <$inet>; while ($line ne "\r\n") { $buffer .= $line; $line = <$inet>; } print "RESP HEADERS:\n$buffer\n"; my $resp = HTTP::Response->parse($buffer); Net::WebSocket::HTTP_R::handshake_consume_response( $handshake, $resp +); my $subscribe = { type => 'subscribe', product_ids => \@prods, channels => ['full'] }; my $payload = encode_json($subscribe); print $payload."\n"; syswrite( $inet, Net::WebSocket::Frame::text->new( payload => $payload, mask => Net::WebSocket::Mask::create(), )->to_bytes() ); #See below about IO::Framed my $iof = IO::Framed->new($inet); $iof->allow_empty_read(); my $parser = Net::WebSocket::Parser->new( $iof ); my $ept = Net::WebSocket::Endpoint::Client->new( parser => $parser, out => $iof, ); $inet->blocking(0); my $sel = IO::Select->new($inet); my $lastseq; my $seq; while(1) { $sel->can_read(); my $frame = $ept->get_next_message(); if($frame) { # we are catching up $frame = $frame->get_payload(); load_frame($frame); } elsif($seq == $lastseq) { # no new data since last iteration if($inet->connected()) { next if $SSL_ERROR == SSL_WANT_READ; if ( $SSL_ERROR == SSL_WANT_WRITE ) { # SSL renegotiation $sel->can_write; next; } # wait for more data usleep(100000); } else { die("WebSocket connection lost!"); } } else { # we are caught up somethinghard(); # catch up again } } sub somethinghard { print "something hard\n"; sleep 1; } sub load_frame { my ($frame) = @_; my $data = decode_json($frame); $lastseq = $seq; $seq = $$data{'sequence'}; print Dumper($data); }
      > I wasn't able to find much about non-blocking sockets with SSL, so hopefully someone can confirm if I did this right.

      For dealing with non-blocking sockets please see the relevant section in the documentation. Please note especially that you cannot simply rely on IO::Select as you did to get notified if new data are available, because data might still be buffered internally in the SSL stack. See the section about common usage errors for details.

        That sample code is where my code changes came from. I'm relying on the websocket module to notify me when data is available, right? I can't read 1 byte at a time like they did because the websocket messages are read with $ept->get_next_message(); Before I made the change, it would get caught in a loop without reading any new data, I'm guessing because SSL needed to be renegotiated like I read in the doc I did find, preventing the websocket module from seeing more data. My change with the select seems to have resolved it.

        I also added a 100ms sleep if there is no message received since the last iteration based on sequence numbers in the messages, but that may never happen because I don't think it goes more than a second without a message.

Re: WebSocket idle
by Anonymous Monk on Jan 03, 2018 at 21:24 UTC
    To me, this sounds like a good application for a second thread.
      If both threads are accessing the same giant hash simultaneously, wouldn't making the hash shared kill performance? I've been through that before.
Re: WebSocket idle
by chris212 (Scribe) on Feb 05, 2018 at 19:58 UTC

    I began using AnyEvent::WebSocket::Client because it supports an idle handler which is exactly what I was looking for and is a higher level implementation with an event loop. However, I can't figure out why the connection keeps closing for no apparent reason. Looking at the packets, it looks like I am initiating the TCP teardown. No idea why.

      no keepalive and/or heartbeat?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1206571]
Approved by stevieb
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (3)
As of 2018-06-21 01:02 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?



    Results (117 votes). Check out past polls.

    Notices?