Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

Forcing to ignore LWP/Net::FTP and use wget/lynx etc

by aufflick (Deacon)
on Sep 28, 2004 at 08:42 UTC ( #394501=perlquestion: print w/replies, xml ) Need Help??

aufflick has asked for the wisdom of the Perl Monks concerning the following question:

At my new client I am working on boxes behind a firewall and an authenticated proxy (which supports browser style ftp over http but not raw ftp proxying.

After installing a dozen pakages by hand, finding dependencies one at a time, I thought "there must be a better way"...

I can download with wget, but the CPAN setup keeps pausing on LWP and Net::FTP until they timeout (which takes a long time) before falling back to wget.

I can try hacking, but I'm guessing there is a real way to do it, but I'm at a loss.

PS: 2 questions in two days - I'm getting less intelligent all the time ;)

  • Comment on Forcing to ignore LWP/Net::FTP and use wget/lynx etc

Replies are listed 'Best First'.
Re: Forcing to ignore LWP/Net::FTP and use wget/lynx etc
by PodMaster (Abbot) on Sep 28, 2004 at 10:20 UTC
    From what I can tell (by scanning for LWP), you need to override sub CPAN::has_usable, something like the following should do
    #!/usr/bin/perl use CPAN(); $CPAN_has_usable = \&CPAN::has_usable; local *CPAN::has_usable = sub { return if $_[1] =~ /^(?:Net|LWP)/; return $CPAN_has_usable->( @_ ); }; do 'path/to/perl/bin/cpan';

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

Re: Forcing to ignore LWP/Net::FTP and use wget/lynx etc
by cfreak (Chaplain) on Sep 28, 2004 at 13:24 UTC

    You can probably solve your problem by using passive FTP. Before you run the CPAN shell just put:

    export FTP_PASSIVE=1

    LWP and Net::FTP should then be able to work around your firewall. You can stick that environment variable in your bashrc as well or if you're using CPAN in a script you should be able to set $ENV{FTP_PASSIVE} = 1 to do the same thing.


    update fixed a typo

    update2 Maybe I should start reading the posts first, I've heard its great. Anyway as dmorgo kindly pointed out, my solution won't do authentication for the proxy, he has the correct answer below.

      He mentioned this is an authenticating proxy. My understanding of authenticating proxies is that this (simply using passive) won't work, because he still needs some way to tell the firewall his userid and password. (See my post below).
Re: Forcing to ignore LWP/Net::FTP and use wget/lynx etc
by dmorgo (Pilgrim) on Sep 28, 2004 at 19:55 UTC
    You are going to get the benefit of my having hacked through this one a few weeks ago! I wish I, then, had been you, now :-). Finally I am going to get some XP points! Woo-hoo! Oops, I mean, Ommmmmmmmm <glances nervously at head monk>.

    Here's what you need for LWP to work with an authenticated proxy (assumes bash):

    export export http_proxy=

    (Change proxyuserid, proxypass, 9090, and the addresses as appropriate.)

    This also assumes there is a call like $ua->env_proxy; in LWP by default (making LWP pay attention to those environment variables by default), which I believe there is.

    Let us know if you figure out Net::FTP. I ended up using ncftpget and wget, and hand editing the CPAN file to make it try those first, as someone else described.

    Here are the settings for ncftpget:

    # ncftpget proxy settings; put in ~/.ncftp/firewall # ftp only firewall-type=2 firewall-user=userid firewall-password=whatever firewall-port=21,localhost,localdomain
    And this is what I have for wget:

    # put in .wgetrc file: http_proxy = ftp_proxy = # ftp untested use_proxy = on proxy-user=userid proxy-passwd=passwd
Re: Forcing to ignore LWP/Net::FTP and use wget/lynx etc
by aufflick (Deacon) on Sep 28, 2004 at 23:01 UTC
    In the words of the current Australian Guinness advertising campain:


    I slightly modified the solution by PodMaster to make my own executable script:

    #!/usr/bin/perl use CPAN(); $CPAN_has_usable = \&CPAN::has_usable; local *CPAN::has_usable = sub { return if $_[1] =~ /^(?:Net|LWP)/; return $CPAN_has_usable->( @_ ); }; CPAN::shell();
    Only wget will work through this proxy, because it virus scans the download and does an http redirect for your "browser" to download the binary. wget is smart enough to follow this convoluted journey.

    To skip lynx and the other command line tools, I hacked my in ~/.cpan/CPAN to say something like:

    'ncftpget' => q[/bin/false], 'ftp' => q[/bin/false], 'lynx' => q[/bin/false],
    Hey - it works.

    Also note that you can set your proxy config in like so:
    'http_proxy' => q[], 'proxy_pass' => q[myPassWord], 'proxy_user' => q[myUser],
    Thankyou all.
      Minor addition to script to avoid an annoying warning:
      use CPAN (); $CPAN_has_usable = \&CPAN::has_usable; local *CPAN::has_usable = sub { return if $_[1] =~ /^(?:Net|LWP)/; return $CPAN_has_usable->( @_ ); }; $ENV{PERL_READLINE_NOWARN} = 1; CPAN::shell();
      It seems that if the shell is started this way rather than directly on the commandline, it complains "Can't ioctl TIOCGETP: Invalid argument" - readline etc. work anyway, so the env variable disables the warning.
Re: Forcing to ignore LWP/Net::FTP and use wget/lynx etc
by martinvi (Monk) on Oct 19, 2004 at 11:56 UTC

    Maybe, I missed a point or two ...

    At my work environment, I've pretty much the same conditions -- i.e. restrictive firewall and an authenticating proxy, which allows only "browser ftp".

    After reading this very thread, I've tweaked (That's down in $YourPerlModulesAreHere/5.8.x/CPAN). When I set ftp_proxy to the same value as http_proxy and put the credentials in the URL, a little happy LWP goes going.

    Here a snippet from

    $CPAN::Config = { # more stuff here ... 'ftp_proxy' => q[http://user:passwd@server:port], 'http_proxy' => q[http://user:passwd@server:port], # ... and here also }

    I hope, anyone -- beside myself -- can made some good use of it.

Re: Forcing to ignore LWP/Net::FTP and use wget/lynx etc
by iburrell (Chaplain) on Sep 28, 2004 at 16:26 UTC
    Can you use HTTP through the proxy? LWP should support that unless your proxy is very weird. Then just use the HTTP CPAN mirrors and don't use the FTP ones.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://394501]
Approved by Corion
Front-paged by Arunbear
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (8)
As of 2019-06-25 11:45 GMT
Find Nodes?
    Voting Booth?
    Is there a future for codeless software?

    Results (105 votes). Check out past polls.