Using File::ReadBackwards with Compressed Files

[Dru]

Hello Monks,

I have compressed log files that I want to give users the option to search from the bottom of the file back. I found File::ReadBackwards which works well for uncompressed log files, but I can't figure out how to use it with my compressed log files. I pefer not to uncompress them first. Can someone show me the best way to do this?

How I'm opening and processing compressed log files:

open (FILE, "zcat $filename |") or die && error($file);

The portion that reads the file starting from the bottom:

# Run this portion if user chooses bottom
 
if ($pos =~ /\bbottom\b|\bb\b/){
 
  my $bw = File::ReadBackwards->new( $file ) 
                or die "can't read $file: $!\n" ;
 
  while( defined( $log_line = $bw->readline ) ) {
 
       print $log_line ;
 
}

Thanks,
Dru

[Roger]

I don't think you can start processing a compressed file from the middle, unless for some simple compression algorithm. Your log file is most likely compressed with compress or gzip that use the LZ77/LZSS sliding window compression algorithm or its variants. The compression algorithm works on building dictionary of words, or phrase book, of repetitive patterns in a look ahead window, and re-encode the matching phrases with a token in the dictionary. And the encoding window moves forward. So I don't it is possible to decode a LZ compressed file backwards. Unfortunately you will have to uncompress the entire file first.

[matija]

I'm afraid you can't do that, Dave. (Sorry, 2001 flashback :-)

An uncompressed logfile will make sense wherever you start to read it, as long as it is at the beginning of a line. However, a compressed file needs to be read from the start. In particular, modern compression alghorhythms change their coding tables as they move through the data they compress. That means you can't make sense of what came later, unless you know what came before.

Even if you found a binary of a zReadBackwardCat it would still have to decompress/read through the whole file in order to understand the end.

Some of the latest compression methods (bzip2?) have resync points scattered through the file, and you might be able to use those to mitigate the worst of it. But I think it would be a lot of work, and it would only work for that particular compression method.

[esskar]

have you tried it like this

my $filepipe = "zcat $filename |";
my $bw = File::ReadBackwards->new($filepipe) or die "can't read $filep
+ipe: $!\n";

but i'm not sure about that...

but anyway... the module probably has to have read in the complete file into memory or temporary store it somewhere completly, so it will probably simpler to decompress it in the first place and than reading it backwards.