Re: What Tools Do You Use With WWW::Mechanize

[cavac]

in reply to What Tools Do You Use With WWW::Mechanize

One problem i encountered with pure WWW::Mechanize was Content-Encoding (compression). At least the version i have installed here announces to the server that it accepts gzip compressed content but doesn't.

Try to use WWW::Mechanize::GZip instead and see if the header problem clears up.

[Limbic~Region]

cavac,
It took me a while to reproduce your problem but I finally encountered a situation where WWW::Mechanize behaves differently than WWW::Mechanize::GZip.

$mech->mirror($url => $file);

With WWW::Mechanize::GZip, the file is properly decoded - not so with WWW::Mechanize. This doesn't mean that WWW::Mechanize doesn't handle compression - this alternative form worked just fine with the same file and WWW::Mechanize:

$mech->get($url, ':content_file' => $file);

[cavac]

I encountered the problem while implementing content compression in Maplat. It happened while while handling forms and downloading dynamically created XML files.

The client was running on Windows with ActivePerl. There *may* have been i misconfiguration or missing package. But i have since reinstalled the windows development server, so i'll have to recreate the circumstances and try if i can come up with a small test system.

It may have been a bug in Maplat as well. But then again it only happened with WWW::Mechanize.

Do you have Compress::Zlib installed? That is - from what i can tell with only a glance at the source - the main difference between WWW::Mechanize and WWW::Mechanize::GZip. The former uses Compress::Zlib when available while the later requires it to be installed.

That said, i'll try to recreate the problem but it might take some time. I'm currently in the middle of preparing a big server upgrade (nice monks are allowed to upgrade to Postgres 9.x while naughty ones have to keep using flat files).

[Limbic~Region]

cavac,
Do you have Compress::Zlib installed?

All versions of ActiveState Perl in recent memory come with both Compress::Zlib and WWW::Mechanize. I develop in a lot of environments under a lot of versions of perl so I can't say what the behavior would be like elsewhere. When I am writing scraping software, it is usually for a client which usually means windows so I had never encountered it before that post.