Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Parse and Summarise Big Brother logs

by McDarren (Abbot)
on Apr 17, 2006 at 13:53 UTC ( #543818=CUFP: print w/ replies, xml ) Need Help??

At my workplace we use Big Brother for Network Monitoring. We have dispensed with the default BB web interface and replaced it with a home-grown Perl/CGI interface...

This is good because:

  • It is more suited to our specific needs. That is, we've added lots of funky CGI options to allow sorting/filtering/etc.
  • It is more "real-time". The default BB interface is basically a static HTML page, which is re-generated every 5 minutes. Whereas our CGI interface is generated every time the page is loaded.

However, the 2nd point above is a double-edged sword. Because with around 15 tests running on 500-odd hosts - that's around 7500 files that must be parsed every time the page is loaded. And when there are 15-20 users constantly hitting the page - things start to get a bit bogged down.

So as a compromise, I decided to write a separate script that would parse all of the BB logs, and summarise them into a single file. And then modify the CGI script so that instead of reading all 7500 logs, it just reads the single summary file.

The summarising script is invoked via crond every minute, and takes less than a second to run. The script is shown below. There is nothing particularly clever about it, but in our case it has proved quite effective. So I thought it may be useful for others.

#!/usr/bin/perl -w # parse_bblogs.pl # Darren - 15th Feb 2006 # # Simple script to parse and summarise Big Brother logs # Would generally be invoked via crond # Will produce a summary of all logs found in $logs_dir # and write to $outfile in the following format: # # hostname:test:status:duration # use strict; use Fcntl qw(:DEFAULT :flock); use Readonly; use Log::Trivial; ################################# # # Adjust the following to suit # regex to match the filename format of the bb logs Readonly my $LOGFILE_MATCH => qr/-\d\./; my $logs_dir = qw( /home/bb/bbvar/logs ); my $outfile = qw( /home/bb/bb/www/bbstatus.current ); my $debug = 0; ################################# # # Should be no need to change anything below here # Create a new logfile each day chomp(my $today = `date +%Y-%m-%d`); my $logfile = qq(/var/log/bb/$0-$today); my $logger = Log::Trivial->new( log_file => "$logfile", log_mode => "single" ); opendir(DIR, $logs_dir) or die "Cannot opendir $logs_dir:$!"; my @files = grep { /$LOGFILE_MATCH/ && -f "$logs_dir/$_" } readdir DIR +; closedir DIR; # Need an exclusive lock on the output file # (Assuming of course, that everybody else is playing by the rules :) sysopen(OUT, $outfile, O_WRONLY | O_CREAT) or die "Cannot open $outfile for writing:$!\n"; flock(OUT, LOCK_EX) or die "Cannot get a lock on $outfile:$!\n"; truncate(OUT, 0) or die "Cannot truncate $outfile:$!\n"; FILE: for my $file (@files) { debug("Processing $file") if $debug; # Because the logs are automagically created by BB, # we can be quite strict about the format we expect to see # ie: hostname.test my ($host, $test) = split(/\./, $file, 2) or $logger->write("WARNING: Skipping unrecognised logfile: $fi +le") and next FILE; open BBLOG, "<", "$logs_dir/$file" or $logger->write("ERROR: I could not open $file: $!") and next FILE; chomp(my @lines = <BBLOG>); # The test status should ALWAYS be the first "word" on the first l +ine # And the status duration should ALWAYS be contained on the 2nd la +st line debug($lines[0], $lines[-2]) if $debug; my ($status) = ($lines[0] =~ /^([a-z]+)/); my ($dur) = ($lines[-2] =~ /Status unchanged in (.*)$/); if (!defined $status && !defined $dur) { $logger->write("WARNING: Skipping malformed logfile: $file"); next FILE; } close BBLOG; # If we get to here, all is good and so we write to the output fil +e print OUT "$host:$test:$status:$dur\n"; } close OUT; sub debug { $logger->write("DEBUG: $_") for @_; }

As always, any pointers for improving the script, making it more efficient, less brain-dead, etc. are most welcome.

Cheers,
Darren :)

Comment on Parse and Summarise Big Brother logs
Download Code
Re: Parse and Summarise Big Brother logs
by chanio (Priest) on Apr 17, 2006 at 21:49 UTC

      How about strftime()?

      use POSIX 'strftime'; print strftime '%Y-%m-%d', localtime();
      HTH,
      Charles

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: CUFP [id://543818]
Approved by kvale
Front-paged by planetscape
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (5)
As of 2014-09-16 02:38 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (155 votes), past polls