Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^3: Old version of LWP::Agent

by Marshall (Prior)
on Apr 27, 2012 at 04:57 UTC ( #967519=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Old version of LWP::Agent
in thread Old version of LWP::Agent

I don't know what kind of debugging info that you expect from WWW::Mechanize? I use WireShark and variants to analyze whether I can get a DNS name resolved, etc.

My internet connect is "fairly reliable", but it will still fail at random unexplainable points - often due to a "barf" at the other end - happens maybe every 3,000 URL's or so. So I just do a re-try which works >99% of the time. Very seldom does one of these things go past one retry and I allow 2 retries.

Logging massive details of all communications in search of an unrepeatable and rare communication failure between you and site X will not help. What are you going to do with that? Sounds like a lot of logging data that will tie your computer into knots while producing no useful actionable information!

If there is some kind of a "bad boy" website (slow, unreliable...whatever), logging that is useful. But you will have that info from the normal functions.


Comment on Re^3: Old version of LWP::Agent
Re^4: Old version of LWP::Agent
by parser (Novice) on Apr 27, 2012 at 14:44 UTC
    Marshall, thank you for your measured and patient responses. I had no idea my post would cause such confusion. I am usually more articulate so this is a new experience for me.

    When I first started using WWW::Mechanize I grew used to setting

    use LWP::Debug qw(+);

    as a simple sanity check to view feedback during my early prototyping phases. Once I need to perform more serious debugging I would use more focused tools such as tamper-data, live-headers, wireshark, etc.

    It sounds like your use case is different than mine. I do not have a requirement to handle 3,000 URIs. I am automating interaction with a limited number of URIs but ones that have fairly complex interaction and formatting (lots of Dojo and Javascript). Having visibility into send and receive traffic (interspersed with my code) at a simple level is useful. Perhaps there is another mechanism I am overlooking besides LWP::Debug.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://967519]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (11)
As of 2015-07-04 16:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (60 votes), past polls