Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

(code) Ignore The Man Behind The system(rsync) Curtain

by ybiC (Prior)
on Mar 01, 2002 at 03:33 UTC ( #148464=sourcecode: print w/ replies, xml ) Need Help??

Category: Utility Scripts
Author/Contact Info ybiC
Description:

Wrapper for rsync, intended for backing up data betwixt a client (Cygwin on Win32|Linux) and a server (Linux).

I looked into File::Rsync module, but it also employs 'exec' calls.   So for now will stick with system call to rsync, for simplicity and for standard-distribution- modules only.

From a Perlish standpoint, this has been a refresher in the use of 'tee' (props to tye and Zaxo), another chance to use the nifty Getopt::Long and swell Pod::Usage modules, use timestamps for logfile names, detect OS type with $^O, use the keen-o filetest operators, sprintf for human-readable date+time, and to write another silly Perl script that's 50% pod.

As always, comments and criticism are wildly welcomed.

Update:
Experimenting with File::Rsync to ease parsing-on-Cygwin woes.
Add parsing code by Zaxo
Minor tweaks to pod
Present runtime in appropriate units (sec, min, hour...)
Handle backups of rsync modules *to* rsync server in addition to *from*


#!/usr/bin/perl -w

# rsync.pl
# pod at tail


$|++;               # stdout hot
use strict;         # avoid d'oh! bugs
require 5;          # for following modules
use Cwd;            # move to particular directory
use Getopt::Long;   # support commandline arguments, options
use Pod::Usage;     # avoid duplicating Usage() in Pod


# EPOCH SECS FOR RUN DURATION:
my $stime = time;

# human-readable time for filenames:
my ($sec,$min,$hour,$mday,$mon,$year) = localtime(time);
my $stamp   = sprintf(
  "%04d%02d%02d%02d%02d",
  $year+1900,$mon+1,$mday,$hour,$min
);



 ## START CONFIG PARAMETERS ##
###############################
my $rsync       = '/usr/bin/rsync -agoptv';
my $tee         = '/usr/bin/tee';
my $d2u         = '/usr/bin/dos2unix';
my $rsyncServer = 'indy';

my $logDir  = '/cygdrive/c/Rsync/logs';
my $allLog  = "$logDir/all$stamp.log";
my $errLog  = "$logDir/err$stamp.log";
my $fileLog = "$logDir/fil$stamp.log";

my $inDir     = '/cygdrive/c/backup';
my @inModules = qw(
  tarballs debs
);
my $outDir     = '/cygdrive/d';
my @outModules = qw(
  data web
);
###############################
 ## END CONFIG PARAMETERS ##



my ( $opt_help, $opt_man, );
GetOptions(
  'help!' => \$opt_help,
  'man!'  => \$opt_man,
);
pod2usage(-verbose => 2) if (defined $opt_man);
pod2usage(-verbose => 1) if (defined $opt_help);




unless (-d $logDir && -w _) {
  print "\nSorry, $logDir doesn't exist,\n",
    "  isn't a directory,\n",
    "  or you don't have write perms there.\n",
    "Please check and correct \$logDir in CONFIG PARAMETERS.\n\n";
  exit;
}



if ($^O eq 'MSWin32') {
  print "\nWin32 cmd.exe doesn't support \'tee\' for logging.\n",
    "Please run from Cygwin bash or cron instead.\n\n";
  exit;
} else {
  open STDOUT, "|$tee $allLog";
  open STDERR, "|$tee $errLog";
}



# HUMAN-READABLE START TIME FOR CONSOLE DISPLAY:
my $htime = sprintf(
  "%04d-%02d-%02d  %02d:%02d:%02d",
  $year+1900,$mon+1,$mday,$hour,$min,$sec
);
print "\n  $htime\n";
print "\n  Starting rsync backup from:\n";
for(@inModules){
  print "    $rsyncServer\:\:$_\n";
}
print "\n  and also rsync backup to:\n";
for(@outModules){
  print "    $rsyncServer\:\:$_\n";
}



 ## START INCOMING XFER ##
###########################
for (@inModules) {
  print "\n == $_ ==\n";
  unless (chdir $inDir) {
    print "Error moving to $inDir: $!";
    next;
  }
  unless (system("$rsync $rsyncServer\:\:$_/ $_/")) {
    print "$?\n";
    next;
  }
  print "\n";
}
###########################
 ## END INCOMING XFER ##



 ## START OUTGOING XFER ##
###########################
for (@outModules) {
  print "\n == $_ ==\n";
  unless (chdir $outDir) {
    print "Error moving to $outDir: $!";
    next;
  }
  unless (system("$rsync $_/ $rsyncServer\:\:$_/")) {
    print "$?\n";
    next;
  }
  print "\n";
}
###########################
 ## END OUTGOING XFER ##



print "\n  Finished rsync backup from:\n";
for(@inModules){
  print "    $rsyncServer\:\:$_\n";
}
print "\n  and also rsync backup to:\n";
for(@outModules){
  print "    $rsyncServer\:\:$_\n";
}



# CALCULATE RUN-TIME:
use constant SECS_PER_MIN  => 60;
use constant SECS_PER_HR   => 3600;
use constant SECS_PER_DAY  => 86400;
use constant SECS_PER_WEEK => 604800;

my $dtime   = time;
my $runSecs = int($dtime-$stime);
my $runTime = $runSecs;
my $runUnit = 'seconds';

if($runSecs > SECS_PER_MIN) {
   $runTime = $runSecs/SECS_PER_MIN;
   $runTime = sprintf("%.0f",$runTime);
   if($runTime == 1.0) {
      $runTime  = 1;
      $runUnit = 'minute';
   } else { $runUnit = 'minutes'; }
}
if($runSecs > SECS_PER_HR) {
   $runTime = $runSecs/SECS_PER_HR;
   $runTime = sprintf("%.1f",$runTime);
   if($runTime == 1.0) {
      $runTime  = 1;
      $runUnit = 'hour';
   } else { $runUnit = 'hours'; }
}
if($runSecs > SECS_PER_DAY) {
   $runTime = $runSecs/SECS_PER_DAY;
   $runTime = sprintf("%.1f",$runTime);
   if($runTime == 1.0) {
      $runTime  = 1;
      $runUnit = 'day';
   } else { $runUnit = 'days'; }
}
if($runSecs > SECS_PER_WEEK) {
   $runTime = $runSecs/SECS_PER_WEEK;
   $runTime = sprintf("%.1f",$runTime);
   if($runTime == 1.0) {
      $runTime  = 1;
      $runUnit = 'week';
   } else { $runUnit = 'weeks'; }
}
print "\n  Runtime  $runTime $runUnit";
print "\n\n";



print
  "  Complete log   $stamp.log\n",
  "  Errors only    $stamp.err\n",
  "  Log directory  $logDir\n",
  "\n";
print <<EOF;
  Cwd          $Cwd::VERSION
  Getopt::Long $Getopt::Long::VERSION
  Pod::Usage   $Pod::Usage::VERSION
  strict       $strict::VERSION
  Perl         $]
  OS           $^O


EOF
;



# PARSE ALL+ERR LOGS FOR NEW+MODIFIED FILES BACKED UP:
close STDERR;
close STDOUT;

system "$d2u $allLog" and die " Error stripping pesky ^M chars from $a
+llLog: $?";
system "$d2u $errLog" and die " Error stripping pesky ^M chars from $e
+rrLog: $?";

# strip trailing slashes from $allLog, $errLog here

open ALL, "< $allLog" or die $!;
open ERR, "< $errLog" or die $!;
open FLE, "> $fileLog" or die $!;

while (<ERR>) {
  {
    local $/ = $_;
    my $diffs = <ALL> || "Alert: '$_' from $errLog not found\n";
    chomp $diffs;
    print FLE $diffs;
  }
}
print FLE while <ALL>;

close FLE or die $!;
close ERR or die $!;
close ALL or die $!;



=head1 NAME

rsync.pl - automate use of rsync for data backups

=head1 DESCRIPTION

 Single purpose wrapper for rsync.

 Intended for backing up data betwixt a
 client (Cygwin on Win32|Linux) and a server (Linux).

 I looked into File::Rsync module, but it also employs
 'exec' calls.  So for now will stick with system call
 to rsync, for simplicity and for standard-distribution-
 modules only.

=head1 SYNOPSIS

C<rsync.pl>

=head1 OPTIONS

 --help
 display Usage, Arguments, and Options

 --man
 display complete man page

=head1 ARGUMENTS

 None: all arguments defined at CONFIG PARAMETERS section of program

=head1 NOTEWORTHY COMMENTS

 Rsync module names cannot contain spaces
 Requires Cygwin (for rsync, bash, tee) to run on Win32
 Source directories must have same name as rsync modules
 Root of backup-to destination directories must be chmod ugo+w
   or rsync.pl will *say* it's backing up files,
   but won't actually write anything to disk!  >8^O
 No trailing / in CONFIG PARAMETER directories nor in rsync mod paths

=head1 BUGS

None that I know of.

=head1 EXAMPLE RSYNC SERVER CONFIG

 # /etc/rsyncd.conf
 hosts allow = 172.16.11.27
 [data]
   comment   = data files backup
   path      = /backup/data
   read only = no
   list      = no
 [web]
   comment   = web files backup
   path      = /backup/web
   read only = no
   list      = no

 # /etc/services
 rsync 873/tcp

 # /etc/inetd.conf
 rsync stream tcp nowait joe /usr/bin/rsync rsyncd --daemon

=head1 UPDATE

 2002-03-4  10:20 CST
  Add explanatory comments
  Generate newly backed-up files log
    strip ^M from all$stamp.log, err$stamp.log
    strip errors from all$stamp.log
  Correct minor tyops
  Research files-backed-up logfile generation

 2002-03-2  16:10 CST
  Explicite path to 'tee'
  Present runtime in appropriate units (sec, min, hour...)
  Handle copy-to and copy-from separately
  Post to PerlMonks
  Filetest for $logDir write perm and directory
    not needed for $localDir as rsync will provide error
  Separate $logDir from $localDir
  Separate '::' from $rsyncServer
  Log STDERR to separate logfile than STDOUT
    to avoid losing head of logfile
  Detect OS, exit if MSWin32 as it doesn't support 'tee'
  Tee STDOUT to logfile and console for all console output
  Getopt and Pod::Usage for --help, --man
  Prettify console info
  Confirm intelligable errors from rsync for
    nonexistant module
    un(known|reachable) host
    host not listening on rsync tcp port 873
  Date+time stamped logfile
  Test for existence of localdir root before writing
  Variablize rsyncServer, localDir, etc

 2002-02-27  22:30 CDT
  Initial working code.

=head1 TODO

  Replace system 'dos2unix' with Perlish approach
  Strip lines ending with / from all$stamp.log, err$stamp.log
  Check for write perms at backup-to destination director(ies)
  Contemplate:
    parsing CONFIG PARAMETERS from external file
    rsync user authentication
    rsync over SSH
  Figure out cron on Cygwin
  Figure out rsyncd on Cygwin

=head1 TESTED

 rsync client  2.4.6      Cygwin 1.36 on Win2Kp 5.00.2195 sp2
 rsync server  2.3.2-1.5  Debian 2.2r5
 Perl          5.006001   Cygwin 1.36 on Win2Kp 5.00.2195 sp2
 Cwd           2.05
 Getopt::Long  2.25
 Pod::Usage    1.14
 strict        1.01

=head1 AUTHOR

ybiC

=head1 CREDITS

 Thanks to tye for tip on using tee to log STDOUT, 
 and Zaxo for reminder to do the same with SDTERR and
 for mondo help with file$stamp.log generation.
 Oh yeah, and to some guy named vroom.

=head1 SEE ALSO

 rsync(1)
 rsyncd(5)
 Perl(1)
 Cygwin

=cut

Comment on (code) Ignore The Man Behind The system(rsync) Curtain
Download Code

Back to Code Catacombs

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: sourcecode [id://148464]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (7)
As of 2014-07-30 05:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (229 votes), past polls