Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight

Retrieving files with cron process

by rbi (Monk)
on Mar 21, 2005 at 21:35 UTC ( #441321=perlquestion: print w/replies, xml ) Need Help??
rbi has asked for the wisdom of the Perl Monks concerning the following question:

Hello, I have a CGI that forks some processing of a set of files to a remote host via ftp.
I have to retrieve by ftp the result of this processing (one file for each file submitted) with a cron process. The list of files to be retrieved is written by the CGI in the $log file below. The processing can be delayed and some submissions might take place while I'm retrieving some results with the cron process (i.e. the CGI execution and the cron process are totally independent). I came up with the following, but I'm definitely not sure if this is robust enough.
Any kind suggestion ?
Thank you.
use strict; use warnings; use Net::FTP; my @files_submitted; my @files_to_retrieve; # Cron procedure to retrieve from a remote host # a set of files that were sent there for processing my ($host,$user,$pass) = ('my_host','user','passwd'); my $log = './logfile.log'; my $tmp_log = './logfile.tmp'; my $working_log = './logfile.wrk'; my $lock = './lockfile.lock'; my $dir = './tmp'; # See if there is a lock still active from # previous execution of this script (it might # be still active, quit in that case) if (-e $lock) { open FH, "<$lock" or die "Can't open $lock: $!\n"; my $pid = <FH>; print "pid $pid \n"; close (FH); exit unless (kill 0 => $pid); } # Create a new lock open FH, ">$lock" or die "Can't open $lock: $!\n"; print FH getppid."\n"; close (FH); # $working_log will contain the list of files to be # retrieved. It can exist if a previous execution # went bad or if the files on the remote host were # not ready when this script was executed the # previous time if (-e $working_log) { open FH, "<$working_log" or die "Can't open $working_log: $!\n"; while (<FH>) { chomp; push @files_to_retrieve,$_; } close (FH); } # See if there are also new files that need # to be retrieved if (-e $log) { # Move the log to avoid interference with files # that could have been submitted between this # execution and the previous one system "mv $log $tmp_log"; open FH, "<$tmp_log" or die "Can't open $tmp_log: $!\n"; while (<FH>) { chomp; push @files_to_retrieve,$_; } } # Check if nothing has to be done goto REMOVE_LOCK unless (@files_to_retrieve); # Loop on the files and process them my @todo = (@files_to_retrieve); my @done; my $ftp = Net::FTP->new($host, Debug => 0); $ftp->login($user,$pass) or die "Cannot login ", $ftp->message; $ftp->cwd($dir); $ftp->binary(); foreach (@files_to_retrieve) { $ftp->get($_) or next; push @done, shift(@todo); } $ftp->quit; # Remove the temporary log system "rm $tmp_log"; # Rewrite the working log if anything went wrong # or some files were not ready if (@todo) { open FH, ">$working_log" or die "Can't open $working_log: $!\n"; foreach (@todo) { print FH $_."\n"; } close (FH); } # Remove lock REMOVE_LOCK: system "rm $lock";

Replies are listed 'Best First'.
Re: Retrieving files with cron process
by perlfan (Curate) on Mar 21, 2005 at 22:09 UTC
    You really have to be more specific. Determine how critical this process is and how bad things would be if a process failed for some reason. You should also determine a list of possible snafu's, and determine which ones you want to handle based on how likely they are to happen, for example:

    - what happens if goes down?
    - what happens if you run out of disk space on locally?

    Also, initially, you may want to generate some sort of email report that lists in detail what happened, and if something went awry.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://441321]
Approved by talexb
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (6)
As of 2018-02-22 11:26 GMT
Find Nodes?
    Voting Booth?
    When it is dark outside I am happiest to see ...

    Results (291 votes). Check out past polls.