Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re^3: Old version of LWP::Agent

by Marshall (Prior)
on Apr 27, 2012 at 04:57 UTC ( #967519=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Old version of LWP::Agent
in thread Old version of LWP::Agent

I don't know what kind of debugging info that you expect from WWW::Mechanize? I use WireShark and variants to analyze whether I can get a DNS name resolved, etc.

My internet connect is "fairly reliable", but it will still fail at random unexplainable points - often due to a "barf" at the other end - happens maybe every 3,000 URL's or so. So I just do a re-try which works >99% of the time. Very seldom does one of these things go past one retry and I allow 2 retries.

Logging massive details of all communications in search of an unrepeatable and rare communication failure between you and site X will not help. What are you going to do with that? Sounds like a lot of logging data that will tie your computer into knots while producing no useful actionable information!

If there is some kind of a "bad boy" website (slow, unreliable...whatever), logging that is useful. But you will have that info from the normal functions.


Comment on Re^3: Old version of LWP::Agent
Re^4: Old version of LWP::Agent
by parser (Novice) on Apr 27, 2012 at 14:44 UTC
    Marshall, thank you for your measured and patient responses. I had no idea my post would cause such confusion. I am usually more articulate so this is a new experience for me.

    When I first started using WWW::Mechanize I grew used to setting

    use LWP::Debug qw(+);

    as a simple sanity check to view feedback during my early prototyping phases. Once I need to perform more serious debugging I would use more focused tools such as tamper-data, live-headers, wireshark, etc.

    It sounds like your use case is different than mine. I do not have a requirement to handle 3,000 URIs. I am automating interaction with a limited number of URIs but ones that have fairly complex interaction and formatting (lots of Dojo and Javascript). Having visibility into send and receive traffic (interspersed with my code) at a simple level is useful. Perhaps there is another mechanism I am overlooking besides LWP::Debug.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://967519]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (15)
As of 2014-07-25 17:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (174 votes), past polls