Re: Old version of LWP::Agent

[Marshall]

I don't know boo about LWP:Agent. I have used LWP::UserAgent - works well. To the best of my knowledge, WWW::Mechanize uses that module also - so it is in current use.

I don't know what "steal, borrow" means. I would just call the LWP::UserAgent routines / inherit them in your own object. I have no idea of what you are doing that LWP::UserAgent or WWW::Mechanize cannot do or why you would want to "steal, borrow" code from these modules rather than use "use" their code. Perhaps you can explain that some more? This idea of using an ancient thing seems weird to me as well as "steal, borrow". I've been using LWP::UserAgent since Perl 5.6 which was quite a while ago.

[parser]

LWP::Debug has long been gutted of any functional code. It is a mere shell so that all of the modules which depend upon it being there do not fail. I am looking for version that contains all of the debug code so that I can leverage it in a Mechanize project. Yes, I can fire up Wireshark but LWP::Debug was useful for its intended purpose.

[Marshall]

I don't know what kind of debugging info that you expect from WWW::Mechanize? I use WireShark and variants to analyze whether I can get a DNS name resolved, etc.

My internet connect is "fairly reliable", but it will still fail at random unexplainable points - often due to a "barf" at the other end - happens maybe every 3,000 URL's or so. So I just do a re-try which works >99% of the time. Very seldom does one of these things go past one retry and I allow 2 retries.

Logging massive details of all communications in search of an unrepeatable and rare communication failure between you and site X will not help. What are you going to do with that? Sounds like a lot of logging data that will tie your computer into knots while producing no useful actionable information!

If there is some kind of a "bad boy" website (slow, unreliable...whatever), logging that is useful. But you will have that info from the normal functions.

[parser]

Marshall, thank you for your measured and patient responses. I had no idea my post would cause such confusion. I am usually more articulate so this is a new experience for me.

When I first started using WWW::Mechanize I grew used to setting

use LWP::Debug qw(+);

as a simple sanity check to view feedback during my early prototyping phases. Once I need to perform more serious debugging I would use more focused tools such as tamper-data, live-headers, wireshark, etc.

It sounds like your use case is different than mine. I do not have a requirement to handle 3,000 URIs. I am automating interaction with a limited number of URIs but ones that have fairly complex interaction and formatting (lots of Dojo and Javascript). Having visibility into send and receive traffic (interspersed with my code) at a simple level is useful. Perhaps there is another mechanism I am overlooking besides LWP::Debug.

[parser]

A big mea cupla is in order.

I misunderstood the comments in the LWP::Debug.pm file as saying the entire module was deprecated. I have implemented the new handlers and they work very well for quick prototyping.