Re: Probelm getting a large compressed tar file over ssh.

[Corion]

You have an error in your script - the command you're sending to the remote machine is not interpolated. You likely want:

$cmd="tar -czf - $rmt_dir";

Depending on how dire your error handling situation is, avoiding Perl might be the most convenient way to resolve your memory issues:

#!/usr/bin/sh
ssh -c "tar -czf - $rmt_dir" >backup.tar.gz

This will run the tar command remotely and output the created .tar.gz file directly to STDOUT, and on the local end will write the output directly into a file instead of buffering it locally in memory. You will need some error checking afterwards though.

[swares]

I used to do this as you sugested but that method requires a trusted host type relationship which I do not have in this environment. I have to conect via ssh to a number of different systems using username / password authentication.

[Corion]

You don't really need a trusted host relationship, you just need passwordless keys. If passwordless keys are impossible, then you will have to use the interactive method. Maybe you can go forward by hacking Net::Ssh::Perl to redirect the STDOUT part of the connection. Looking into the source of Net::SSH::Perl::SSH1, there seem to be handlers like SSH_SMSG_STDOUT_DATA and replacing the default handler with something that doesn't accumulate the string might help:

# Original code
sub cmd {
    ...
    unless ($ssh->handler_for(SSH_SMSG_STDOUT_DATA)) {
        $ssh->register_handler(SSH_SMSG_STDOUT_DATA,
            sub { $ssh->{_cmd_stdout} .= $_[1]->get_str });
    }
    ...
}

I would try to supply my own callback like this:

my $ssh = Net::SSH::Perl->new(...);
open my $outfh, ">", $filename
  or die "Couldn't create '$filename' : $!";
binmode $outfh;
$ssh->register_handler('stdout', sub { print $outfh $_[1]; });

[0xbeef]

If passwordless keys are at all possible, you could also consider File::Remote.

This is from the example:

use File::Remote qw(:replace);
   # read from a remote file
   open(REMOTE, "host:/remote/file") or die $!;
   print while (<REMOTE>);
   close(REMOTE);

   # writing a local file still works!
   open(LOCAL, ">>/local/file");
   print LOCAL "This is a new line.\n";
   close(LOCAL);

   # Read and write whole files
   my @file = $remote->readfile("host:/remote/file");
   $remote->writefile("/local/file", @file);

0xbeef

[science_gone_bad]

I was following this thread, and I never saw anything about the GNU tar portion.

I've had issues in the past where GNU tar chokes if the tar file created exceeds 2GB in size. I don't know if this has been fixed, but I was seeing this issue as late as Sept of 2004.
Maybe...