Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Re^3: Old version of LWP::Agent

by Marshall (Abbot)
on Apr 27, 2012 at 04:57 UTC ( #967519=note: print w/replies, xml ) Need Help??

in reply to Re^2: Old version of LWP::Agent
in thread Old version of LWP::Agent

I don't know what kind of debugging info that you expect from WWW::Mechanize? I use WireShark and variants to analyze whether I can get a DNS name resolved, etc.

My internet connect is "fairly reliable", but it will still fail at random unexplainable points - often due to a "barf" at the other end - happens maybe every 3,000 URL's or so. So I just do a re-try which works >99% of the time. Very seldom does one of these things go past one retry and I allow 2 retries.

Logging massive details of all communications in search of an unrepeatable and rare communication failure between you and site X will not help. What are you going to do with that? Sounds like a lot of logging data that will tie your computer into knots while producing no useful actionable information!

If there is some kind of a "bad boy" website (slow, unreliable...whatever), logging that is useful. But you will have that info from the normal functions.

Replies are listed 'Best First'.
Re^4: Old version of LWP::Agent
by parser (Novice) on Apr 27, 2012 at 14:44 UTC
    Marshall, thank you for your measured and patient responses. I had no idea my post would cause such confusion. I am usually more articulate so this is a new experience for me.

    When I first started using WWW::Mechanize I grew used to setting

    use LWP::Debug qw(+);

    as a simple sanity check to view feedback during my early prototyping phases. Once I need to perform more serious debugging I would use more focused tools such as tamper-data, live-headers, wireshark, etc.

    It sounds like your use case is different than mine. I do not have a requirement to handle 3,000 URIs. I am automating interaction with a limited number of URIs but ones that have fairly complex interaction and formatting (lots of Dojo and Javascript). Having visibility into send and receive traffic (interspersed with my code) at a simple level is useful. Perhaps there is another mechanism I am overlooking besides LWP::Debug.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://967519]
[atcroft]: .oO(Then there is the effect if a site changes their timezone, such as when the International Date Line was moved by the purchase of Alaska by the US from Russia in 1867, or several places (I cannot recall off-hand) that moved from one side of the Date
[atcroft]: Line to the other recently....)
[atcroft]: .oO(Then again, you also have to be careful if you had it to the database, and be aware of any assumptions it makes (such as SQLite assuming Gregorian calendar and a day of exactly 86400 seconds)...)

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (4)
As of 2017-04-29 04:53 GMT
Find Nodes?
    Voting Booth?
    I'm a fool:

    Results (531 votes). Check out past polls.