Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

automated file move

by scopey (Novice)
on Dec 17, 2007 at 19:40 UTC ( #657495=perlquestion: print w/ replies, xml ) Need Help??
scopey has asked for the wisdom of the Perl Monks concerning the following question:

Due to hdd space constraints, I'd like to write a script that will automatically move freshly generated output files to another disk with lots of free space. This script will run on a 2nd PC. All PCs are Windows. So far, the best idea I could think of involves checking file attributes, namely the size element of stat, and keeping track of its state so that I know when the file has stopped growing and therefore it's time to move it. Like this:
my $outputf = 'C:\temp\output_log.txt'; # The following is here just for a reminder: my ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime, +$blksize,$blocks) = (0,1,2,3,4,5,6,7,8,9,10,11,12); while (1) { my @stat = stat $outputf; print "\n"; for (@stat) {print "$_\t"} sleep 1; }
Size seems the only file attribute that always changes and stops changing when the output file is complete. But there must be a better way than this...?? Thank you.

Comment on automated file move
Download Code
Re: automated file move
by pc88mxer (Vicar) on Dec 17, 2007 at 21:01 UTC
    Under Linux you can use the fuser command to determine if another process has the file open. That would be another useful piece of information to use in your determination of whether or not the log file is still being written to.

    I'm sure there is something equivalent for Windows. You can also use the fact that some Windows commands don't let you rename files which are open or rename directories which have open files under them them. Don't know what they are, but I run into that problem all the time using the Windows Explorer.

Re: automated file move
by RaduH (Scribe) on Dec 17, 2007 at 21:19 UTC
    The problem is a bit under-specified. The cycle to test a resource (read value, sleep, read value, sleep, etc.) is correct. Well, assuming that your print inside foreach is just a placeholder and you actually do some processing there such as getting the size and doing something with the file if size ... (your criteria here).

    What is not clear is what you mean by "the file has stopped growing". I hope you don't conclude "hard disk full" when this happens... The semantics and dynamics of log file access are not known in your problem description. Based on what I read, the only thing that can be improved it fine-tuning the sleep interval ("stopped growing" means two consecutive reads yield the same size? or size doesn't change in 5 minutes? it is not clear ...). If you want the file to be moved when it reaches a certain size (popular option in handling logs), you can fine-tune your sleep based on the file growth speed (oh, well, not always easy to guesstimate, depending on the use scenario ...) and just move the file the first time it exceeds your size limit (that would be checked for in the foreach body.

    Hard to give more details/opinions with only the info you provided, but your approach is good.

Re: automated file move
by aquarium (Curate) on Dec 17, 2007 at 23:21 UTC
    unless this is for theoretical purposes only as an excercise in perl....data is normally moved off to separate storage in chunks (such as entire directory trees) with due regard given to access to that data from the application(s) that need it. It doesn't make sense to continue directory contents onto another server with such manual (and easily breakable) code.
    you may be able to leverage MS DFS technology for unifying UNC access or other filesystem/clustering technology
    the hardest line to type correctly is: stty erase ^H
Re: automated file move
by megaurav2002 (Monk) on Dec 18, 2007 at 00:14 UTC
    I know something called Rdiff Backup that takes incremental backups. You can investigate it to see if suits your requirements.

    cheers
    Gaurav

    "Wisdom begins in wonder" - Socrates, philosopher
Re: automated file move
by pfaut (Priest) on Dec 18, 2007 at 00:33 UTC

    I suspect you want to wait until the writing process closes the file before you copy it and are using the fact that the file isn't growing anymore to assume this has occurred. Since this is Windows, you could probably take advantage of file locking instead. If you attempt to open the file without allowing other accessors, the open should fail unless the other process has closed it. You could retry with a delay until successful.

    Another option might be to use a utility like RoboCopy which should be available from one of Microsoft's download sites if it's not on your system already. It should detect locked files and exit with a status indicating whether it succeeded or not.

    90% of every Perl application is already written.
    dragonchild
Re: automated file move
by poolpi (Hermit) on Dec 18, 2007 at 10:05 UTC
Re: automated file move
by scopey (Novice) on Dec 20, 2007 at 21:24 UTC
    Thanks to everyone for all the thoughtful replies, and taking time to sort out my somewhat lackluster description of the task. :) I tried several methods, including attempting to open the file for writing, but found what seems to be a fairly simple (and most reliable) solution using Perl's 'move' command. The following script can be run as is, provided the user sets the proper input/output directories and desired filetype. On my PC, this script runs with no noticeable effect on CPU usage. Any comments appreciated.
    #!/usr/local/bin/perl use strict; use warnings; use File::Basename; # Gives us fileparse() use File::Path; # Gives us mkpath use File::Copy; # Gives us move ############################################################### # NOTE: Set these variables according to your paths and desired # filetype. # # Set input\output paths, filetype, and sleep time. my $inputd = 'C:\temp'; my $outputd = 'D:\test\output'; my $filetype = 'pcm'; my $sleeptime = 1; # seconds to wait between directory reads ############################################################### # Disable print buffer. $| = 1; # Print message to user. print "\nAutomated File Move Utility"; print "\n\nThis script is attempting to move all $filetype files:"; print "\nsource: $inputd\ndestination: $outputd"; print "\n\nPress Ctrl+C to stop...\n\n"; # Repeat indefinitely. while (1) { # Create output directory if it doesn't exist already. if (!-e $outputd) { mkpath ($outputd, 0) or die "\nError - unable to create $outputd: $!\n\n"; } # Get files (full path and name). my @infiles = glob ("$inputd\\*.$filetype"); # Form hash that relates input and output files. my %outfiles; foreach my $inputf (@infiles) { my $basename = fileparse ($inputf, $filetype); $outfiles{$inputf} = "$outputd\\$basename"."$filetype"; } # This block just provides feedback for testing purposes. if (0) { print "\nINPUT:"; foreach (@infiles) {print "\n$_"} print "\nOUTPUT:"; foreach (keys %outfiles) {print "\n$outfiles{$_}"} } # Wait for some time. sleep $sleeptime; # Attempt to move each input file. foreach my $inputf (@infiles) { next if (! move $inputf, $outfiles{$inputf}); print "\nMoved $inputf to $outfiles{$inputf}"; } }

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://657495]
Approved by Old_Gray_Bear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (6)
As of 2014-11-27 06:00 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (180 votes), past polls