Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

LWP oddity

by adamsj (Hermit)
on Nov 12, 2002 at 23:13 UTC ( #212437=perlquestion: print w/ replies, xml ) Need Help??
adamsj has asked for the wisdom of the Perl Monks concerning the following question:

So we're using LWP to pull data off a server by pretending we're a browser and going through a proxy--or we were, until firewall issues cut us off. They've added us to the allow list and should be getting through, but we're not.

"Okay," I says to myself, "if we can't go through the proxy 'cause it thinks we're it instead of us, I can just take the proxy setting out, and the request will just whiz right through."

Instead, now the script dies while I'm stepping through it in the debugger. This bothers me.

my $list_request = HTTP::Request->new(GET => $source_dir); my $agent = LWP::UserAgent->new; $agent->protocols_allowed(['ftp', 'http']); #$agent->proxy('ftp' => 'http://www.nunya.biz:8080'); my $list_response = $agent->request($list_request); my @lines = grep (s!^<A HREF="^\w+($loaddate)($file_pattern)([\w.]*).* +$!$2$3!, (split(/\n/,$list_response->content)) );

Now, with the proxy setting taken out, it fails in the debugger on the next line.

I'm puzzled--any help?

adamsj

They laughed at Joan of Arc, but she went right ahead and built it. --Gracie Allen

Comment on LWP oddity
Download Code
Re: LWP oddity
by dws (Chancellor) on Nov 13, 2002 at 00:22 UTC
    They've added us to the allow list and should be getting through, but we're not.

    Until you get clarity on how the "allow list" works, you're going to suffer on this one. Are they allowing by IP address (in which case little you do at your end is going to make a difference), or are they allowing by browser type (which you can forge)?

    And are you really sure you want to be proxying 'ftp' and not 'http'?

    I've had little luck using FTP through firewalls, unless passive FTP is supported by the remote server. YMMV.

      It does look odd, doesn't it? But I recall tinkering with the proxy statement for quite a while before getting a working combination with
      $agent->proxy('ftp' => 'http://www.nunya.biz:8080');
      Now that the proxy server has been added to the allow list, I'm getting my files just fine. From my boss's point of view, the problem is fixed--but I still want to know why I'm silently dying when I remove the proxy statement!

      adamsj

      They laughed at Joan of Arc, but she went right ahead and built it. --Gracie Allen

Re: LWP oddity
by trs80 (Priest) on Nov 13, 2002 at 00:40 UTC
    You can turn on LWP debugging:
    require LWP::Debug; LWP::Debug::level('+');
    if you haven't already.
    Update: Forgot the require LWP::Debug the first time
      I hadn't thought to set LWP::Debug (and did you know that Perl and LWP doesn't deal with LWP::Debug at all?), but now I have, and am getting much the same result when the proxy statement is removed. Without proxy, the output from LWP::Debug is:
      LWP::UserAgent::request: () LWP::UserAgent::send_request: GET ftp://user:password@10.10.10.10/outg +oing/ LWP::UserAgent::_need_proxy: Not proxied LWP::UserAgent::send_request: ftp URLs are among LWP::UserAgent=HASH(0 +x83ae480)'s allowed protocols (ftp http) LWP::MediaTypes::read_media_types: Reading media types from /my/lib/pe +rl5/site_perl/5.6.1/LWP/media.types LWP::Protocol::ftp::request: () LWP::Protocol::MyFTP::new: ()
      and here the debugger dies, along with the script.

      With the proxy statement left in, I get:

      LWP::UserAgent::request: () LWP::UserAgent::send_request: GET ftp://user:password@10.10.10.10/outg +oing/ LWP::UserAgent::_need_proxy: Proxied to http://www.nunya.biz:8080 LWP::UserAgent::send_request: http URLs are among LWP::UserAgent=HASH( +0x83d28b0)'s allowed protocols (ftp http) LWP::Protocol::http::request: () LWP::Protocol::collect: read 315 bytes LWP::Protocol::collect: read 4096 bytes LWP::Protocol::collect: read 4096 bytes LWP::Protocol::collect: read 2954 bytes LWP::Protocol::collect: read 1460 bytes LWP::Protocol::collect: read 660 bytes LWP::Protocol::collect: read 32 bytes LWP::UserAgent::request: Simple response: OK
      and the debugger stays alive and ready to proceed.

      On the off-chance this was an artifact of the debugger, I ran without the -d flag and sprinkled in some print statements. This didn't produce any change--LWP::Debug gave the same output and I died in the same place.

      So at this point, I'm slightly stumped. I suppose my next move is either to give up and say "What the heck do I care? It works, doesn't it?" or step laboriously through the LWP code itself to see where it dies.

      I'm certainly open to a better idea than either of those two.

      Afterthought: I'm running 5.6.1. LWP is version 5.63.

      adamsj

      They laughed at Joan of Arc, but she went right ahead and built it. --Gracie Allen

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://212437]
Approved by Theseus
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (8)
As of 2014-10-25 19:57 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (148 votes), past polls