Beefy Boxes and Bandwidth Generously Provided by pair Networks Ovid
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

lwp proxy problem

by dannoura (Pilgrim)
on Apr 27, 2004 at 15:36 UTC ( #348540=perlquestion: print w/ replies, xml ) Need Help??
dannoura has asked for the wisdom of the Perl Monks concerning the following question:

hi,

I'm trying to work through my ISP provider's proxy, but for some reason every time I issue a request with it I get the ISP's homepage, which is obviously not the page I requested. I tried two scripts. The first is:

use LWP::Simple qw(&getstore $ua);

$ua->proxy(http => 'http://www.tau.ac.il/tau.pac');

my $url='http://www.yahoo.com';
my $file='test.html';
print "success" if (getstore($url, $file));

[download]

The second is:

use LWP::UserAgent;

open WFH, ">test.html" or die($!); 
 
my $ua = LWP::UserAgent->new;
$ua->agent("Mozilla/5.0");
$ua->env_proxy;
$ua->proxy('http', 'http://www.tau.ac.il/tau.pac');
$ua->timeout(10);

my $response = $ua->get('http://search.cpan.org/');


if ($response->is_success) {
    print WFH $response->content;
    } else {
    die $response->status_line;
    }

[download]

Any ideas as to the reason?

-----------------------------------

Any comments about coding style are welcome.

Comment on lwp proxy problem
Select or Download Code
Re: lwp proxy problem
by cyocum (Curate) on Apr 27, 2004 at 15:46 UTC

    This is a completely off the cuff comment but the file you point to in $ua->proxy(http => 'http://www.tau.ac.il/tau.pac'); is an automatic configuration script, which is used in web browsers. I do not think that LWP supports that.

[reply]
[d/l]
Re: lwp proxy problem
by bronto (Priest) on Apr 27, 2004 at 15:52 UTC

    I downloaded the file using wget, getting:

    Read more... (2 kB)

    It is easy to read, so I think that once you know your IP address you can pick a proxy from the list by hand

    Or you can write a script yourself to parse the pac file, at the moment I have no time to sketch one, sorry :-(

    Update: see HTTP::ProxyAutoConfig

    Update: try this and tell me if it works. It seems I can't use that proxy, so I couldn't make any test myself, apart from checking that it parses and runs correctly.

    Read more... (1000 Bytes)

    Update: actually, that won't work, since you need to parse the return value from FindProxy and pass it to $ua->proxy only if it matches /PROXY: host:port/ (after cleaning up the PROXY: parts, of course). Unfortunately, I am having no luck in making this module work (the FindProxyForURL function doesn't get defined). I'll update the above script as soon as I make sense of the module...

    Update: Ok, I checked with ptkdb and saw that:

    • the module tries to parse the function contained in the pac file (javascript) to convert it to Perl
    • the parser in the subroutine is very buggy, and probabily expects a certain format for the PAC file, that is: is not general enough to parse any pac
    • the converted function is finally eval'ed, but no check is done on the $@ variable. Therefore, the user doesn't know if the pac failed to parse until he tries to use the FindProxy* functions and sees them fail because FindProxyForURL doesn't exist

    Possible solutions:

    • write a bug report for the module and wait for the author to patch his module
    • write a better parsing routine, or ask for one (I can't write one, I am a JavaScript illiterate :-)
    • rewrite the parser routine using one of the JavaScript modules from CPAN. This one, for example, uses the SpiderMonkey engine from Mozilla, and I expect that it will know how to parse the PAC

    Sorry, I can't help further :-(

    Ciao!
    --bronto

    The very nature of Perl to be like natural language--inconsistant and full of dwim and special cases--makes it impossible to know it all without simply memorizing the documentation (which is not complete or totally correct anyway).
    --John M. Dlugosz
[reply]
[d/l]
[select]

      Thanks for your help. It's not working, though. I get two errors:

      Useless use of a constant in void context at C:/perl/site/lib/HTTP/Pro
+xyAutoConfig.pm line 367.
Undefined subroutine &HTTP::ProxyAutoConfig::FindProxyForURL called at
+ C:/perl/site/lib/HTTP/ProxyAutoConfig.pm line 131.

      [download]

      -----------------------------------

      Any comments about coding style are welcome.
[reply]
[d/l]
Re: lwp proxy problem
by matija (Priest) on Apr 27, 2004 at 15:56 UTC
    As cyocum correctly deduced, your problem is that you're setting the proxy to the configuration file.

    That file is Javascript, which is supposed to be executed by the browser on a URL-by-URL basis to determine which proxy to use for which URL.

    LWP does not (and probably never will) support that.

    You need to read the file, understand it, and select a proxy from it yourself: $ua->proxy('http','proxyslb.tau.ac.il:8080'); for example.

[reply]
[d/l]

Back to Seekers of Perl Wisdom

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://348540]
Approved by Old_Gray_Bear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (22)
As of 2013-05-21 18:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best material for plates (tableware) is:









    Results (443 votes), past polls