monger has asked for the wisdom of the Perl Monks concerning the following question:
Greetings Fellow Monks, and various other Clerics,
I've written a small script to parse through a directory of log files looking for a signature. The directory always has four files. I am searching for the signature in all files, then dumping the matching lines out to a file, called vpnwarn.out. What I need to do each time before hand is look to see if vpnwarn.out exists. If it does, I need to move it to vpnwarn.old and unlink vpnwarn.out. Then the script will run. Here's the pertinent parts of the script:
use strict;
use warnings;
use File::Copy "mv";
use Data::Dumper;
no strict 'refs';
my @names;
my $file;
my $name;
my $regex = 'WARNING';
#Variable to hold path to the log directory
my $dir = "H:/vpn/";
my $out = "H:/vpn/vpnwarn.out";
my $old = "H:/vpn/vpnwarn.old";
if (-s $out) {
mv ($out, $old)";
unlink ($out)";
}
What ends up happening is that the vpnwarn.out file grows and grows, along with the .old file. I'd originally tried to use Logfile::Rotate and Logfile::Cern, but neither would compile on Cygwin. I'm approaching this in this fashion so daily, we can have a diff report of the two files, seeing any new activity.
Short of help fixing either module for Cygwin, does anyone have any suggestions on how to clear the .out for overwriting?
Thanks,
Monger
Re: Simple Log Rotate Problem
by meetraz (Hermit) on Nov 13, 2003 at 21:51 UTC
|
update: it looks like you are trying to move the $out file to the $old file, but you're not deleting the $old file first. Then you are trying to unlink() the $out file, but if the mv() succeeded, it would fail anyway.
Try this:
my $dir = "H:/vpn/";
my $out = "H:/vpn/vpnwarn.out";
my $old = "H:/vpn/vpnwarn.old";
if (-e $old) {
unlink($old) || die ("Could not unlink: $!");
}
if (-e $out) {
mv ($out, $old) || die ("Could not mv: $!");
}
# create and write new $out here
| [reply] [Watch: Dir/Any] [d/l] |
|
So what you're really saying is your unlink() call is not succeeding? try this:
if (-s $out) {
mv ($out, $old) || die ("Could not mv: $!");
unlink ($out) || die ("Could not unlink: $!");
}
And see why the unlink() call is failing.
The unlink is failing. Sorry for not pointing that out earlier. The return is:
Can't unlink file: No such file or directory at ./vpnwarn.pl line 28.
However, the file does exist. If I delete both of the log files, .old and .out, and run the script, the first time it succeeds, the second it fails with the above error.
Hope that sheds some more light.
Monger
| [reply] [Watch: Dir/Any] |
|
my $dir = "H:/vpn/";
my $out = "H:/vpn/vpnwarn.out";
my $old = "H:/vpn/vpnwarn.old";
if (-e $old) {
unlink($old) || die ("Could not unlink: $!");
}
if (-e $out) {
mv ($out, $old) || die ("Could not mv: $!");
}
# create and write new $out here
| [reply] [Watch: Dir/Any] [d/l] |
|
Re: Simple Log Rotate Problem
by particle (Vicar) on Nov 13, 2003 at 22:09 UTC
|
when you move the file, doesn't that delete it?
from the File::Copy docs
* The "move" function also takes two parameters: the current nam
+e and
the intended name of the file to be moved. If the destination
already exists and is a directory, and the source is not a
directory, then the source file will be renamed into the direc
+tory
specified by the destination.
If possible, move() will simply rename the file. Otherwise, it
copies the file to the new location and deletes the original.
+If an
error occurs during this copy-and-delete process, you may be l
+eft
with a (possibly partial) copy of the file under the destinati
+on
name.
You may use the "mv" alias for this function in the same way t
+hat
you may use the "cp" alias for "copy".
of course unlink is failing!
| [reply] [Watch: Dir/Any] [d/l] |
|
I prefer to cp/truncate instead of deleting. That way you do not have to worry about restarting the service that has the file open for logdumping.
on commandline it looks like this:
rm logfile.5
mv logfile.4 logfile.5
mv logfile.3 logfile.4
mv logfile.2 logfile.3
mv logfile.1 logfile.2
cp logfile logfile.1
cat /dev/null > logfile
its about the same thing in perl except you use File::Copy's copy and move and open OUTFILE, ">logfile"; close OUTFILE;
to truncate
| [reply] [Watch: Dir/Any] [d/l] [select] |
Re: Simple Log Rotate Problem
by duff (Parson) on Nov 13, 2003 at 23:42 UTC
|
Another approach you could take would be include the date in the filename, so instead of writing to vpnwarn.out all the time you could write to vpnwarn-11-12-2003.out (using POSIX::strftime() or something to format the filename). Then you could use a separate program to clean up these files when their number becomes too great.
It's strange that your files continue to grow if your code closely follows what you've posted here.
| [reply] [Watch: Dir/Any] |
Re: Simple Log Rotate Problem
by shockme (Chaplain) on Nov 13, 2003 at 21:56 UTC
|
Following the move, whatever service is writing to the file will need to be restarted as well, for example:
system("/etc/init.d/syslog restart") == 0 or ...
I've seen situations where the application would continue writing to the old file until the service was restarted. Odd but true, and it might explain some of what you're seeing.
If you can, check /etc/logrotate.conf or any of the files in /etc/logrotate.d, and you'll probably see where certain services are being restarted following the rotation.
If things get any worse, I'll have to ask you to stop helping me.
| [reply] [Watch: Dir/Any] [d/l] |
|
Following the move, whatever service is writing to the file will need to be restarted as well, for example:
system("/etc/init.d/syslog restart") == 0 or ...
I've seen situations where the application would continue writing to the old file until the service was restarted. Odd but true, and it might explain some of what you're seeing.
If you can, check /etc/logrotate.conf or any of the files in /etc/logrotate.d, and you'll probably see where certain services are being restarted following the rotation.
Thanks for the idea. It could be the issue, but Perl is what is creating these files. They are snapshots of the real log files. The script will run, creating vpnwarn.out. Upon next execution, after Perl has exited, it fails with
Can't unlink file: No such file or directory at ./vpnwarn.pl line 28. (see reply to earlier post). If the previous instance of Perl is mucking with it, let me know.
Also, there is no logrotate here. It's cygwin on a Doze box.
Monger
| [reply] [Watch: Dir/Any] |
|
dingdingdingdingdingdingding
The magic word is cygwin, which means that you do have a unix-y environment. shockme's node touches on exactly what is going on. If a process has a filehandle open when you move the old vpnwarn.out, that process will continue writing to the same file, now named vpnwarn.old. If another process comes along and connects to the new vpnwarn.out, then both files will continue to grow. (tested under cygwin)
It gets worse: If the (still in use) vnpwarn.old file is unlinked while the filehandle is still open, it can continue to grow without even showing up when you list the directory. You can chew up a whole hard drive's capacity that way.
You say that perl is creating these files. When logging, try to follow an "open, write, close" cycle as much as possible. This not only flushes your output (in case you forgot to turn off buffering), but avoids the "phantom menace" of unlinked but expanding files.
My parents just came back from a planet where the dominant life form had no
bilateral symmetry, and all I got was this stupid F-Shirt.
| [reply] [Watch: Dir/Any] |
|
Since the log file is on an "H:" drive, this appears to be a Win32 system. Unlike Unix, on Win32 you can't move a file that's in use, so I don't think this would be in issue there. (and the logrotate service doens't exist)
| [reply] [Watch: Dir/Any] |
|
I see your point (and based on monger's reply, it appears that you are correct). However, I've had situations where a *nix directory was mounted to a Windows box, and scripts would run on the Windows box rather than the *nix machine. Weird? Yes, but I didn't set it up ...
So, I didn't assume that H: precluded the existence of a *nix environment.
If things get any worse, I'll have to ask you to stop helping me.
| [reply] [Watch: Dir/Any] |
|
|