Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re: Dealing with binary data and WWW::Mechanize and encoding stuff

by davidrw (Prior)
on Dec 07, 2008 at 15:19 UTC ( #728724=note: print w/ replies, xml ) Need Help??


in reply to Dealing with binary data and WWW::Mechanize and encoding stuff

Taking advantage of LWP::UserAgent's "Handlers" (and WWW::Mechanize is a proper subclass of LWP::UserAgent) might give you better access to the content. And using IO::Compress::Gzip for better dealing with the content a piece at a time.

use IO::Compress::Gzip; my $compressed; my $z; $mech->add_handler( response_header => sub { my($response, $ua, $h) = @_; $response->{default_add_content} = 0; $z = new IO::Compress::Gzip \$compressed or die; } ); $mech->add_handler( response_data => sub { my($response, $ua, $h, $data) = @_; print $z $data or die $!; return 1; } ); $mech->add_handler( response_done => sub { my($response, $ua, $h) = @_; close $z or die $!; } ); $mech->get($pdf_url); warn length $mech->content; # 0 cause of the 'default_add_content' se +tting warn length $compressed;
Note that $z->print() and $z->close() work too.
Note that LWP::UserAgent has a remove_handler method, too, in case this $mech object has to go do other stuff.


Also, does IO::Compress::Gzip::gzip handle it any better directly from $mech->content?

(side note) Can (potentially) save a mem copy of the content by instead passing $mech->content_ref to Compress::Zlib::memGzip


Comment on Re: Dealing with binary data and WWW::Mechanize and encoding stuff
Select or Download Code
Re^2: Dealing with binary data and WWW::Mechanize and encoding stuff
by friedo (Prior) on Dec 07, 2008 at 15:34 UTC
    Cool. I had forgotten about LWP::UserAgent's callback features. (In fact, there's all sorts of goodies in LWP::UA that one doesn't remember if one looks only at the Mech docs.)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://728724]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (13)
As of 2015-07-03 06:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (48 votes), past polls