|Problems? Is your data what you think it is?|
Solved: Uncompress streaming gzip on the fly in LWP::UserAgent/WWW::Mechanizeby Your Mother (Bishop)
|on Dec 18, 2018 at 22:15 UTC||Need Help??|
Your Mother has asked for the wisdom of the Perl Monks concerning the following question:
Update: New and cleaned up code posted as a CUFP in Streaming Market Quotes from Ally Invest.
I include only the bare bones because I tried something like 20 different things without success and I'm embarrassed. :( Non-streaming requests are working perfectly with approximately this code. The endpoint for this code is a streaming quote and trade ticker. The URL shown just asks for the info for AAPL. The compression—well, the ongoing, memory sane handling of it—is the only problem I have.
The # $data is where the handling, or a core part of it, is missing.
The closest I get streaming to work is gunzip from IO::Uncompress::Gunzip. It works fine but it's a standalone conversion which is goofy/impossible for an "infinite" stream of data; append, gunzip, append, gunzip… It must be uncompressed in toto. So a non-starter but I could see it works fine at least. Whatever the solution, I guess it will require IO handle truncation as it goes.
I played with opening a pipe to gunzip(1), I tried every variation of Compress::Zlib I could think but wasn't able to get any data out or apparently even write/uncompress the data properly. I tried everything I could figure out with IO::Uncompress::Gunzip too. A few PM and SO nodes looked promising but I didn't find anything that seemed to apply directly or fix it. I even tried PerlIO::gzip. I'm just stick stupid on this. Maybe one of the attempts was close but I have no idea which, and, again, I'm embarrassed to share any of the *many*.
The doc page https://www.ally.com/api/invest/documentation/streaming/. I tried to read the sample streaming code for other languages but there is no gzip handling surfaced in them. If they work, it's hidden in the libraries used.
On the plus side, the WWW::OAuth was trivial to mix in and works great for this.
Thanks for looking!
Caveat for anyone doing the same kind of thing, the $mech response accumulates so you still have to clear it out with $response->content(undef) or it slowly, perpetually grows…