Forcing CPAN.pm to ignore LWP/Net::FTP and use wget/lynx etc

[aufflick]

At my new client I am working on boxes behind a firewall and an authenticated proxy (which supports browser style ftp over http but not raw ftp proxying.

After installing a dozen pakages by hand, finding dependencies one at a time, I thought "there must be a better way"...

I can download with wget, but the CPAN setup keeps pausing on LWP and Net::FTP until they timeout (which takes a long time) before falling back to wget.

I can try hacking CPAN.pm, but I'm guessing there is a real way to do it, but I'm at a loss.

PS: 2 questions in two days - I'm getting less intelligent all the time ;)

[PodMaster]

From what I can tell (by scanning CPAN.pm for LWP), you need to override sub CPAN::has_usable, something like the following should do

#!/usr/bin/perl

use CPAN();

$CPAN_has_usable = \&CPAN::has_usable;

local *CPAN::has_usable = sub {
    return if $_[1] =~ /^(?:Net|LWP)/;
    return $CPAN_has_usable->( @_ );
};

do 'path/to/perl/bin/cpan';

[cfreak]

You can probably solve your problem by using passive FTP. Before you run the CPAN shell just put:

export FTP_PASSIVE=1

LWP and Net::FTP should then be able to work around your firewall. You can stick that environment variable in your bashrc as well or if you're using CPAN in a script you should be able to set $ENV{FTP_PASSIVE} = 1 to do the same thing.

HTH

update fixed a typo

update2 Maybe I should start reading the posts first, I've heard its great. Anyway as dmorgo kindly pointed out, my solution won't do authentication for the proxy, he has the correct answer below.

[dmorgo]

He mentioned this is an authenticating proxy. My understanding of authenticating proxies is that this (simply using passive) won't work, because he still needs some way to tell the firewall his userid and password. (See my post below).

[dmorgo]

You are going to get the benefit of my having hacked through this one a few weeks ago! I wish I, then, had been you, now :-). Finally I am going to get some XP points! Woo-hoo! Oops, I mean, Ommmmmmmmm <glances nervously at head monk>.

Here's what you need for LWP to work with an authenticated proxy (assumes bash):

export ftp_proxy=proxyuserid:[e-mail]:21
export http_proxy=http://proxyuserid:[e-mail]:9090

(Change proxyuserid, proxypass, 9090, and the addresses as appropriate.)

This also assumes there is a call like $ua->env_proxy; in LWP by default (making LWP pay attention to those environment variables by default), which I believe there is.

Let us know if you figure out Net::FTP. I ended up using ncftpget and wget, and hand editing the CPAN file to make it try those first, as someone else described.

Here are the settings for ncftpget:

# ncftpget proxy settings; put in ~/.ncftp/firewall
# ftp only
firewall-type=2
firewall-host=hostname.com
firewall-user=userid
firewall-password=whatever
firewall-port=21
firewall-exception-list=.foo.com,localhost,localdomain

And this is what I have for wget:

# put in .wgetrc file:
http_proxy = http://wwwproxy.foo.com:9090/
ftp_proxy = ftpproxy.foo.com:21 # ftp untested
use_proxy = on
proxy-user=userid
proxy-passwd=passwd

[aufflick]

In the words of the current Australian Guinness advertising campain:

Brilliant!

I slightly modified the solution by PodMaster to make my own executable script:

#!/usr/bin/perl

use CPAN();

$CPAN_has_usable = \&CPAN::has_usable;

local *CPAN::has_usable = sub {
    return if $_[1] =~ /^(?:Net|LWP)/;
    return $CPAN_has_usable->( @_ );
};

CPAN::shell();

Only wget will work through this proxy, because it virus scans the download and does an http redirect for your "browser" to download the binary. wget is smart enough to follow this convoluted journey.

To skip lynx and the other command line tools, I hacked my MyConfig.pm in ~/.cpan/CPAN to say something like:

  'ncftpget' => q[/bin/false],
  'ftp' => q[/bin/false],
  'lynx' => q[/bin/false],

Hey - it works.
Also note that you can set your proxy config in MyConfig.pm like so:

  'http_proxy' => q[10.10.10.10:8000],
  'proxy_pass' => q[myPassWord],
  'proxy_user' => q[myUser],

Thankyou all.

[aufflick]

Minor addition to script to avoid an annoying warning:

use CPAN ();

$CPAN_has_usable = \&CPAN::has_usable;

local *CPAN::has_usable = sub {
    return if $_[1] =~ /^(?:Net|LWP)/;
    return $CPAN_has_usable->( @_ );
};

$ENV{PERL_READLINE_NOWARN} = 1;

CPAN::shell();

It seems that if the shell is started this way rather than directly on the commandline, it complains "Can't ioctl TIOCGETP: Invalid argument" - readline etc. work anyway, so the env variable disables the warning.

[martinvi]

Maybe, I missed a point or two ...

At my work environment, I've pretty much the same conditions -- i.e. restrictive firewall and an authenticating proxy, which allows only "browser ftp".

After reading this very thread, I've tweaked Config.pm (That's down in $YourPerlModulesAreHere/5.8.x/CPAN). When I set ftp_proxy to the same value as http_proxy and put the credentials in the URL, a little happy LWP goes going.

Here a snippet from Config.pm:

$CPAN::Config = {

# more stuff here ...

'ftp_proxy' => q[http://user:passwd@server:port],
'http_proxy' => q[http://user:passwd@server:port],

# ... and here also

}

I hope, anyone -- beside myself -- can made some good use of it.

[iburrell]

Can you use HTTP through the proxy? LWP should support that unless your proxy is very weird. Then just use the HTTP CPAN mirrors and don't use the FTP ones.