Re: Dealing with binary data and WWW::Mechanize and encoding stuff

[davidrw]

Taking advantage of LWP::UserAgent's "Handlers" (and WWW::Mechanize is a proper subclass of LWP::UserAgent) might give you better access to the content. And using IO::Compress::Gzip for better dealing with the content a piece at a time.

use IO::Compress::Gzip;
my $compressed;
my $z;
$mech->add_handler( response_header => sub {
  my($response, $ua, $h) = @_;
  $response->{default_add_content} = 0;
  $z = new IO::Compress::Gzip \$compressed or die;
} );
$mech->add_handler( response_data => sub {
  my($response, $ua, $h, $data) = @_;
  print $z $data or die $!;
  return 1;
} );
$mech->add_handler( response_done => sub {
  my($response, $ua, $h) = @_;
  close $z or die $!;
} );

$mech->get($pdf_url);
warn length $mech->content;  # 0 cause of the 'default_add_content' se
+tting
warn length $compressed;

Note that $z->print() and $z->close() work too.
Note that LWP::UserAgent has a remove_handler method, too, in case this $mech object has to go do other stuff.


Also, does IO::Compress::Gzip::gzip handle it any better directly from $mech->content?

(side note) Can (potentially) save a mem copy of the content by instead passing $mech->content_ref to Compress::Zlib::memGzip

[friedo]

Cool. I had forgotten about LWP::UserAgent's callback features. (In fact, there's all sorts of goodies in LWP::UA that one doesn't remember if one looks only at the Mech docs.)