Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Simple Log Rotate Problem

by monger (Friar)
on Nov 13, 2003 at 21:37 UTC ( [id://306936]=perlquestion: print w/replies, xml ) Need Help??

monger has asked for the wisdom of the Perl Monks concerning the following question:

Greetings Fellow Monks, and various other Clerics, I've written a small script to parse through a directory of log files looking for a signature. The directory always has four files. I am searching for the signature in all files, then dumping the matching lines out to a file, called vpnwarn.out. What I need to do each time before hand is look to see if vpnwarn.out exists. If it does, I need to move it to vpnwarn.old and unlink vpnwarn.out. Then the script will run. Here's the pertinent parts of the script:
use strict; use warnings; use File::Copy "mv"; use Data::Dumper; no strict 'refs'; my @names; my $file; my $name; my $regex = 'WARNING'; #Variable to hold path to the log directory my $dir = "H:/vpn/"; my $out = "H:/vpn/vpnwarn.out"; my $old = "H:/vpn/vpnwarn.old"; if (-s $out) { mv ($out, $old)"; unlink ($out)"; }
What ends up happening is that the vpnwarn.out file grows and grows, along with the .old file. I'd originally tried to use Logfile::Rotate and Logfile::Cern, but neither would compile on Cygwin. I'm approaching this in this fashion so daily, we can have a diff report of the two files, seeing any new activity. Short of help fixing either module for Cygwin, does anyone have any suggestions on how to clear the .out for overwriting? Thanks, Monger

Replies are listed 'Best First'.
Re: Simple Log Rotate Problem
by meetraz (Hermit) on Nov 13, 2003 at 21:51 UTC

    update: it looks like you are trying to move the $out file to the $old file, but you're not deleting the $old file first. Then you are trying to unlink() the $out file, but if the mv() succeeded, it would fail anyway.

    Try this:

    my $dir = "H:/vpn/"; my $out = "H:/vpn/vpnwarn.out"; my $old = "H:/vpn/vpnwarn.old"; if (-e $old) { unlink($old) || die ("Could not unlink: $!"); } if (-e $out) { mv ($out, $old) || die ("Could not mv: $!"); } # create and write new $out here
      So what you're really saying is your unlink() call is not succeeding? try this: if (-s $out) { mv ($out, $old) || die ("Could not mv: $!"); unlink ($out) || die ("Could not unlink: $!"); } And see why the unlink() call is failing.

      The unlink is failing. Sorry for not pointing that out earlier. The return is: Can't unlink file: No such file or directory at ./vpnwarn.pl line 28. However, the file does exist. If I delete both of the log files, .old and .out, and run the script, the first time it succeeds, the second it fails with the above error.

      Hope that sheds some more light.

      Monger

        Try this:
        my $dir = "H:/vpn/"; my $out = "H:/vpn/vpnwarn.out"; my $old = "H:/vpn/vpnwarn.old"; if (-e $old) { unlink($old) || die ("Could not unlink: $!"); } if (-e $out) { mv ($out, $old) || die ("Could not mv: $!"); } # create and write new $out here
Re: Simple Log Rotate Problem
by particle (Vicar) on Nov 13, 2003 at 22:09 UTC

    when you move the file, doesn't that delete it?

    from the File::Copy docs

    * The "move" function also takes two parameters: the current nam +e and the intended name of the file to be moved. If the destination already exists and is a directory, and the source is not a directory, then the source file will be renamed into the direc +tory specified by the destination. If possible, move() will simply rename the file. Otherwise, it copies the file to the new location and deletes the original. +If an error occurs during this copy-and-delete process, you may be l +eft with a (possibly partial) copy of the file under the destinati +on name. You may use the "mv" alias for this function in the same way t +hat you may use the "cp" alias for "copy".

    of course unlink is failing!

    ~Particle *accelerates*

      I prefer to cp/truncate instead of deleting. That way you do not have to worry about restarting the service that has the file open for logdumping. on commandline it looks like this:
      rm logfile.5 mv logfile.4 logfile.5 mv logfile.3 logfile.4 mv logfile.2 logfile.3 mv logfile.1 logfile.2 cp logfile logfile.1 cat /dev/null > logfile
      its about the same thing in perl except you use File::Copy's copy and move and open OUTFILE, ">logfile"; close OUTFILE; to truncate


      -Waswas
Re: Simple Log Rotate Problem
by duff (Parson) on Nov 13, 2003 at 23:42 UTC

    Another approach you could take would be include the date in the filename, so instead of writing to vpnwarn.out all the time you could write to vpnwarn-11-12-2003.out (using POSIX::strftime() or something to format the filename). Then you could use a separate program to clean up these files when their number becomes too great.

    It's strange that your files continue to grow if your code closely follows what you've posted here.

Re: Simple Log Rotate Problem
by shockme (Chaplain) on Nov 13, 2003 at 21:56 UTC
    Following the move, whatever service is writing to the file will need to be restarted as well, for example:
    system("/etc/init.d/syslog restart") == 0 or ...
    I've seen situations where the application would continue writing to the old file until the service was restarted. Odd but true, and it might explain some of what you're seeing.

    If you can, check /etc/logrotate.conf or any of the files in /etc/logrotate.d, and you'll probably see where certain services are being restarted following the rotation.

    If things get any worse, I'll have to ask you to stop helping me.

      Following the move, whatever service is writing to the file will need to be restarted as well, for example: system("/etc/init.d/syslog restart") == 0 or ... I've seen situations where the application would continue writing to the old file until the service was restarted. Odd but true, and it might explain some of what you're seeing. If you can, check /etc/logrotate.conf or any of the files in /etc/logrotate.d, and you'll probably see where certain services are being restarted following the rotation.

      Thanks for the idea. It could be the issue, but Perl is what is creating these files. They are snapshots of the real log files. The script will run, creating vpnwarn.out. Upon next execution, after Perl has exited, it fails with Can't unlink file: No such file or directory at ./vpnwarn.pl line 28. (see reply to earlier post). If the previous instance of Perl is mucking with it, let me know.

      Also, there is no logrotate here. It's cygwin on a Doze box.

      Monger

        dingdingdingdingdingdingding

        The magic word is cygwin, which means that you do have a unix-y environment. shockme's node touches on exactly what is going on. If a process has a filehandle open when you move the old vpnwarn.out, that process will continue writing to the same file, now named vpnwarn.old. If another process comes along and connects to the new vpnwarn.out, then both files will continue to grow. (tested under cygwin)

        It gets worse: If the (still in use) vnpwarn.old file is unlinked while the filehandle is still open, it can continue to grow without even showing up when you list the directory. You can chew up a whole hard drive's capacity that way.

        You say that perl is creating these files. When logging, try to follow an "open, write, close" cycle as much as possible. This not only flushes your output (in case you forgot to turn off buffering), but avoids the "phantom menace" of unlinked but expanding files.


        My parents just came back from a planet where the dominant life form had no
        bilateral symmetry, and all I got was this stupid F-Shirt.
      Since the log file is on an "H:" drive, this appears to be a Win32 system. Unlike Unix, on Win32 you can't move a file that's in use, so I don't think this would be in issue there. (and the logrotate service doens't exist)
        I see your point (and based on monger's reply, it appears that you are correct). However, I've had situations where a *nix directory was mounted to a Windows box, and scripts would run on the Windows box rather than the *nix machine. Weird? Yes, but I didn't set it up ...

        So, I didn't assume that H: precluded the existence of a *nix environment.

        If things get any worse, I'll have to ask you to stop helping me.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://306936]
Approved by meetraz
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others imbibing at the Monastery: (4)
As of 2024-03-29 09:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found