Get specific entries from a file which is always growing

[ashok.g]

Hi Folks,

Do anyone have an idea on how to achieve the following task using a shell/perl script?

Lets say some data is being written forever to File1. I need to get some specific data from File1 (say lines containing word 'IDENTIFIER') and write to File2 always.

The question seems to be simple but I think it is tricky and need some good time to get the solution.

Any help on this is highly appreciated. Thanks in advance.

-- Ashok

[wind]

Check out File::Tail

[Eliya]

A quick test shows that something like the following simple approach seems to work rather well (on a Unix file system, where you can read a file that's still open for writing):

open my $fh, "<", "growingfile" or die;

while (1) {
    while (<$fh>) {
        print if /IDENTIFIER/;
    }
    sleep 1;
}

I.e. you just resume reading after you've reached the (current) end of file.

The tricky part is probably when IDENTIFIER has only been partially read before EOF is encountered. To handle this case, you'd need to concat the last read incomplete record after resuming...  (Whether this is relevant at all would depend on the buffering mode used to write the file (i.e. unbuffered, line-buffered, fully/block-buffered), and what your input records are (e.g. lines).)

[NetWallah]

Assuming this is linux (use cygwin otherwise), shell commands are made for this:

  grep "IDENTIFIER" File1 > File2
  tail -f File1 | grep "IDENTIFIER" >> File2

[ashok.g]

This doesn't seem to be working. I am not getting any data in the File2 using these commands.

[Eliya]

I suppose you're talking about the second command. The first grep command should definitely put something in File2, as long as there's some occurrences of "IDENTIFIER" in File1 at the time you run the command.

The "problem" with the second command line is that when redirecting grep's output to a file, the stream will be fully buffered. This means that nothing will be written to File2 before the buffer (typically 4K in size) is full, or tail closes its pipe to grep (the latter won't happen when tail is running in "follow" (-f) mode).   Depending on how much output grep produces with your data, filling the buffer may take a while...

Whether the buffering is a problem depends on your expectations. If you want File2 to be updated immediately with every newly found occurrence, then the above shell command is not the right tool for the purpose. In this case, you could write the tail/grep combo yourself in Perl (as already suggested), in which case you do have control over buffering (i.e. you can flush data immediately).

[BrowserUk]

Do you control the process that is writing file1?