moving old files

[pbaumgar]

Hello everyone: I'm working on a script that will go through my apache httpd log directory and gzip and move log files older than 24 hours to an archive directory. Since I'm new to perl programming I was hoping others could offer advice on "best practices" or "best way to do it" on the perl code I've written so far. Here's my script so far:

#!/usr/bin/perl -w
use strict;
use File::Copy;

my $log_dir = "/etc/httpd/logs/2006";
my $archive_dir = "/etc/httpd/logs/archive";

#open logs directory and get files older than 24 hours
opendir(LOGS, $log_dir) 
    or die "Directory could not be opened: $!";
my @files = grep { -M "$log_dir/$_" > 1 } readdir(LOGS);
closedir(LOGS)
    or die "Directory could not be closed: $!";

#check each file in logs directory
my @to_move;
foreach my $file (@files) {
    $file eq '.' || $file eq '..'
    or push @to_move, $file;
}

#perform file test; compress file; move file
foreach my $file (@to_move) {
    if (! -l $file)
    {
    chdir $log_dir
        or die "Can't chdir to $log_dir: $!";
    system "gzip $file";
    move "$file.gz", $archive_dir;
    } else {
    die "Can't move $file: $!";
    }
}

[Paladin]

#check each file in logs directory
my @to_move;
foreach my $file (@files) {
    $file eq '.' || $file eq '..'
    or push @to_move, $file;
}

The code in this section is a bit hard to read. A more readable way would be something like:

#check each file in logs directory
my @to_move;
foreach my $file (@files) {
    next if $file =~ /^\.\.?$/;
    push @to_move, $file;
}

Or better yet, remove this part, and add it to the readdir earlier.

my @files = grep { -M "$log_dir/$_" > 1 && !/^\.\.?$/ } readdir(LOGS);

A few other things:
• Use the LIST form of system when you are able to: system "gzip", $file
• chdir to the log dir once at the start of the script instead of once for each file processed
• In the else block you use $! but there is no guarantee that it will be set to a valid value before hand. You know at that point that the file is a symbolic link, so just print that.
• Nit Pick: You seem to use 2 different indent styles, one for the foreach and another for the if (You indent the body of one, but not the other). It usually helps to be consistant.

[rhesa]

I'd use shell scripting for this, something like this:

log_dir  = "/etc/httpd/logs/2006"
arch_dir = "/etc/httpd/logs/archive"
find $log_dir -mtime +0 -type f -exec gzip {} \; -exec mv {}".gz" $arc
+h_dir \;

[davidrw]

or use logrotate .. very nice & powerful (see man page)

[brd]

It would be easier to use something like cronolog (http://www.cronolog.org). Logrotate would also work, after moving the files remember to HUP Apache though. Sorry this is not a Perl solution, but there is no reason to reinvent the wheel. :)

[superfrink]

++ brd. I was just going to recommend chronolog.

Chronolog uses the Apache feature of logging to another process instead of to a file. This means you do not have to do:


- rename all of the log files
- HUP apache
- then archive the old log fles however you like

The best part is apache does not stop reponding for a short while during the HUP. I used to admin for a company that ran several hundred virtual hosts per apache install. That HUP can be noticable.

Chronolog takes care of creating and renaming files for you as time goes by. I use something like the following:

  LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}
+i\"" combined
  ErrorLog "|/usr/local/sbin/cronolog /log/web/domain.com/%Y-%m-%d_err
+or.log"
  CustomLog "|/usr/local/sbin/cronolog /log/web/domain.com/%Y-%m-%d_ac
+cess.log" combined

This setup gives me one access log and one error log per calendar day. I think you can setup log files per hour or minute if you want to.

[brd]

The only problem I have noticed with using Cronolog is that it adds two processes everytime it is used. If you have lots of virtual hosts and a cronolog for the ErrorLog & CustomLog that adds four processes per VirtualHost.
 
Edit: It seems that only on Apache 1.3 does apache create two processes for a piped logging program. It forks a shell and then runs cronolog. I'm seeing this in an Apache 1.3 server:

13311  ??  I      0:00.00 /bin/sh -c /usr/local/sbin/cronolog /usr/log
+s/httpd/%Y/%m/access_log
13314  ??  I      0:00.36 /usr/local/sbin/cronolog /usr/logs/httpd/%Y/
+%m/access_log

vs. on Apache 2.0:

29394  ??  I      0:00.04 /usr/local/sbin/cronolog /usr/logs/httpd/%Y/
+%m/access_log

[pbaumgar]

Hello again,
I've decided to use Cronolog for Apache logging. I have cronolog creating logs with like this: %m-%Y/access_log. So it will seperate all the logs my months. What is the best way to handle the month change in my original script? I'd like at the end of them month to gzip the entire directory and dump it in the archive directory. Do I use a file/directory test to check the timestamp on it?


my $log_dir = "/etc/httpd/logs/2006";
my $archive_dir = "/etc/httpd/logs/archive";