Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Re: Dealing with binary data and WWW::Mechanize and encoding stuff

by davidrw (Prior)
on Dec 07, 2008 at 15:19 UTC ( #728724=note: print w/replies, xml ) Need Help??

in reply to Dealing with binary data and WWW::Mechanize and encoding stuff

Taking advantage of LWP::UserAgent's "Handlers" (and WWW::Mechanize is a proper subclass of LWP::UserAgent) might give you better access to the content. And using IO::Compress::Gzip for better dealing with the content a piece at a time.
use IO::Compress::Gzip; my $compressed; my $z; $mech->add_handler( response_header => sub { my($response, $ua, $h) = @_; $response->{default_add_content} = 0; $z = new IO::Compress::Gzip \$compressed or die; } ); $mech->add_handler( response_data => sub { my($response, $ua, $h, $data) = @_; print $z $data or die $!; return 1; } ); $mech->add_handler( response_done => sub { my($response, $ua, $h) = @_; close $z or die $!; } ); $mech->get($pdf_url); warn length $mech->content; # 0 cause of the 'default_add_content' se +tting warn length $compressed;
Note that $z->print() and $z->close() work too.
Note that LWP::UserAgent has a remove_handler method, too, in case this $mech object has to go do other stuff.

Also, does IO::Compress::Gzip::gzip handle it any better directly from $mech->content?

(side note) Can (potentially) save a mem copy of the content by instead passing $mech->content_ref to Compress::Zlib::memGzip

Replies are listed 'Best First'.
Re^2: Dealing with binary data and WWW::Mechanize and encoding stuff
by friedo (Prior) on Dec 07, 2008 at 15:34 UTC
    Cool. I had forgotten about LWP::UserAgent's callback features. (In fact, there's all sorts of goodies in LWP::UA that one doesn't remember if one looks only at the Mech docs.)

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://728724]
[Lady_Aleena]: I don't want to have to do: alpha_menu($hash, undef, $type);
[Lady_Aleena]: Or what Discipulus said.
[shmem]: Lady_Aleena: geany supports ctags.
[Discipulus]: a good compromise can be my ($need, $opts_ref) = @_ a scalar and an hash reference
[Discipulus]: see you monks!
[Lady_Aleena]: shmem, let me get this sub rewritten, then I will look into how to use ctags in geany. Deal? 8)
[shmem]: Discipulus: yeah, that might eventually prepare the path for OO ;-)
[Lady_Aleena]: See you, Discipulus.
[shmem]: Lady_Aleena: that's up to you. I only wanted to show you a path that might be more comforting than command line grep ;-)
[Lady_Aleena]: shmem, I don't think any of my modules could be converted to OO. They are too procedural.

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (8)
As of 2017-04-27 12:25 GMT
Find Nodes?
    Voting Booth?
    I'm a fool:

    Results (506 votes). Check out past polls.