Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re^3: Streaming Market Quotes from Ally Invest

by talexb (Canon)
on Jan 31, 2019 at 22:30 UTC ( #1229211=note: print w/replies, xml ) Need Help??


in reply to Re^2: Streaming Market Quotes from Ally Invest
in thread Streaming Market Quotes from Ally Invest

I just walked to the grocery store and back while thinking about it -- and I suspect you are correct. I'm able to log in and navigate using Mech, even though I see some Javascript on the page, but I think this download is asynchronous, thus it relies on JS, thus Mech can't catch it. Ugh.

I'd love to be able to try something like headless Chrome, but I don't think I'll be able to persuade my client to take that route. The low-tech alternative would be to have someone manually download the file every couple of days, and then upload it to the local server (which I manage). Thanks for your reply, and your original post -- it's always good to read about success stories.

Alex / talexb / Toronto

Thanks PJ. We owe you so much. Groklaw -- RIP -- 2003 to 2013.

  • Comment on Re^3: Streaming Market Quotes from Ally Invest

Replies are listed 'Best First'.
Re^4: Streaming Market Quotes from Ally Invest
by Your Mother (Bishop) on Feb 01, 2019 at 05:39 UTC

    It is also possible to capture the entire exchange (the dev panel(s) can do it) and see exactly what the JS is doing for headers and cookies and such. It should be pretty easy to then emulate the requests with plain HTTP in Perl code. It might be fragile but it's probably not too hard. Write a test and put it on a nightly cron to alert you if/when it breaks.

      This is a brilliant idea, and I should have thought of it myself. Thanks!

      Alex / talexb / Toronto

      Thanks PJ. We owe you so much. Groklaw -- RIP -- 2003 to 2013.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1229211]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (6)
As of 2019-07-17 22:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?