Streaming Market Quotes from Ally Invest

[Your Mother]

This service requires you to have an account with Ally. Their investment API is new. I don't work for them or have any professional relationship but they are often mentioned as the best online bank in the US, I've been a customer since they were GMAC (20 years), and they really came through for me once so I have no reservations in recommending them and sharing some code to use their services.

I had trouble getting streaming to work which was embarassing because it's supposed to be the part of Perl I'm good at. :P pmqs, haukex bliako, kschwab, and vr all stepped up to help and solved my trouble. My new trouble is a websocket client—a cat to skin later—which led me back to Mojolicious. I knew it had some server support so I figured the example code might show client code. I was pleasantly shocked to see it supports it completely via its own client Mojo::UserAgent. All hail sri!

I was even happier to see the client supports gzip content, even in streams. Might be more confusing code for some but for me it was much easier to follow. I have hesitated to move some of my personal code/practices to Mojo but this is probably the shove I needed.

Since the problem is now solved with both libraries, I figured I should share some of the code here. Both rely on the terrific WWW::OAuth. There is a fair bit of identical boilerplate for the arguments and environment.

WWW::Mechanize (LWP::UserAgent)

#!/usr/bin/env perl
use 5.10.0;
use strictures;
use WWW::Mechanize; # LWP::UserAgent is almost the same here.
use WWW::OAuth;
use Compress::Zlib;

my @symbols = grep /\A[A-Z.]+\z/, @ARGV;
die "Give a list of symbols; e.g., AAPL GHII AMZN XOM DIS PBF BABA JD 
+AMD VOO\n"
    unless @symbols;

my $sym = join ",", @symbols;

die "Missing ENV values: ALLY_CLIENT_ID ALLY_CLIENT_SECRET ALLY_TOKEN 
+ALLY_TOKEN_SECRET\n"
    unless $ENV{ALLY_CLIENT_ID}
    and $ENV{ALLY_CLIENT_SECRET}
    and $ENV{ALLY_TOKEN}
    and $ENV{ALLY_TOKEN_SECRET};

my $oauth = WWW::OAuth->new(
    client_id => $ENV{ALLY_CLIENT_ID},
    client_secret => $ENV{ALLY_CLIENT_SECRET},
    token => $ENV{ALLY_TOKEN},
    token_secret => $ENV{ALLY_TOKEN_SECRET} );

my $mech = WWW::Mechanize->new( autocheck => undef );
$mech->add_handler( request_prepare => sub { $oauth->authenticate($_[0
+]) } );

my $gunzip = inflateInit( WindowBits => 16 + MAX_WBITS )
            or die "Cannot create a inflation stream\n";

$mech->add_handler (
    response_data => sub {
    my ( $response, $ua, $h, $data ) = @_;
    $response->content(undef); # Else will append.
    my ( $buffer, $status ) = $gunzip->inflate($data);
    die "zlib error: $status" if length $status;
    say $buffer;
    });

$mech->get("https://stream.tradeking.com/v1/market/quotes?symbols=$sym
+");

__END__

Mojo::UserAgent

#!/usr/bin/env perl
use 5.10.0;
use strictures;
use Mojo::UserAgent;
use WWW::OAuth;

my @symbols = grep /\A[A-Z.]+\z/, @ARGV;
die "Give a list of symbols; e.g., AAPL GHII AMZN XOM DIS PBF BABA JD 
+AMD VOO\n"
    unless @symbols;

my $sym = join ",", @symbols;

die "Missing ENV values: ALLY_CLIENT_ID ALLY_CLIENT_SECRET ALLY_TOKEN 
+ALLY_TOKEN_SECRET\n"
    unless $ENV{ALLY_CLIENT_ID}
    and $ENV{ALLY_CLIENT_SECRET}
    and $ENV{ALLY_TOKEN}
    and $ENV{ALLY_TOKEN_SECRET};

my $oauth = WWW::OAuth->new(
    client_id => $ENV{ALLY_CLIENT_ID},
    client_secret => $ENV{ALLY_CLIENT_SECRET},
    token => $ENV{ALLY_TOKEN},
    token_secret => $ENV{ALLY_TOKEN_SECRET} );

my $ua = Mojo::UserAgent->new( max_response_size => 0 ); # Stream mean
+s no max.
$ua->on( start => sub { $oauth->authenticate( $_[1]->req ) } ); # OAut
+h all requests.

my $tx = $ua->build_tx( GET => "https://stream.tradeking.com/v1/market
+/quotes?symbols=$sym" );

$tx->res->content
    ->unsubscribe("read")
    ->on( read => sub {
        my ( $content, $bytes ) = @_;
        say $bytes;
      });

$tx = $ua->start($tx);

__END__

[talexb]

I'm writing a script that logs on to a website, does some navigation, clicks on a button that generates and then downloads a CSV, and I'm wondering if the add_handler method will do what I need to capture this CSV. I had a look on WWW::Mechanize, but didn't see any information on this method.

Right now I'm calling the content method after the click, and that just gives me the web page and not the CSV that arrives a few seconds later. Is there more information about this method?

[Your Mother]

I suspect(?) there is some JS involved if doing a $mech->click on the download link is not getting the CSV file. That kind of thing should "just work" and if it doesn't, putting in handlers won't make any difference. You might check corion's WWW::Mechanize::Chrome or WWW::Mechanize::Firefox to see if it works more like the browser.

[talexb]

I just walked to the grocery store and back while thinking about it -- and I suspect you are correct. I'm able to log in and navigate using Mech, even though I see some Javascript on the page, but I think this download is asynchronous, thus it relies on JS, thus Mech can't catch it. Ugh.

I'd love to be able to try something like headless Chrome, but I don't think I'll be able to persuade my client to take that route. The low-tech alternative would be to have someone manually download the file every couple of days, and then upload it to the local server (which I manage). Thanks for your reply, and your original post -- it's always good to read about success stories.