Re: Store log file content from EOF till final occurrence of timestamp

[Athanasius]

Hello jayu_rao,

First, if this code:

my $TIMESTAMP;
...
for (@dates) {
     $TIMESTAMP = $_->[0]->strftime("%Y-%m-%d %R");
}

does what I think it does, it can be written more simply and efficiently without the loop:

my $TIMESTAMP = $dates[-1][0]->strftime("%Y-%m-%d %R");

Update: Changed $dates[-1]->strftime to $dates[-1][0]->strftime.

Now, let’s look at your second challenge:

The 2nd log file may not have the exact timestamp (in the seconds field) as in log1 file.

You don’t say where the strftime method comes from, but if it follows the same conventions as the POSIX::strftime::GNU module, then the %R conversion specification produces “[t]he time in 24-hour notation (%H:%M)” only — so $TIMESTAMP is guaranteed not to contain seconds. (If you want seconds, you have to use %T.)

Now to the first challenge. Try something like this:

my @log_final;
my $last = 0;

while (my $log_line = $log_bckwards->readline) {
    if ($log_line =~ /$TIMESTAMP/) {
        $last = 1;
    }
    elsif ($last) {
        last;
    }
    unshift @log_final, $log_line;
}

Note that in the code you have shown, the capture and /g modifier in your regex do nothing.

Hope that helps,

[1nickt]

I think the OP wanted all lines before the last instance of the TIMESTAMP match, including non-matching ones.

Better maybe as:

my @log_final;
my $watching_for_last_match = 0;
#  ^ Bonus! This var will give a count of 
#    entries within the timestamp at the end,
#    compare with scalar(@log_final) for stats

while (my $log_line = $log_bckwards->readline) {
  if ( $watching_for_last_match ) {    
    if ($log_line !~ /$TIMESTAMP/) {
      last;
    }
  } else {
    if ($log_line =~ /$TIMESTAMP/) {
      $watching_for_last_match++;
    }
  }        
  unshift @log_final, $log_line;
}

[Athanasius]

Hello 1nickt,

I think the OP wanted all lines before the last instance of the TIMESTAMP match, including non-matching ones.

Yes, exactly, although since we are reading the file backwards, we should perhaps say: all lines from the first TIMESTAMP match (inclusive) to the end of the file.

Leaving aside the added line count, it seems to me that your code is logically identical to mine. Am I missing something?

---

But in reading your post it occurs to me that there is a logic error in both of our approaches: what if the TIMESTAMP never has an exact match? The OP is, apparently, confident that there will always be a match, but a more robust design would allow for the possibility that there may not be:

my @log_final;

while (my $log_line = $log_bckwards->readline) {
    my $current_timestamp = extract_timestamp($log_line);
    last if compare_timestamps($current_timestamp, $TIMESTAMP) < 0;
    unshift @log_final, $log_line;
}

— with the implementation of extract_timestamp() and compare_timestamps() being left as the proverbial exercise for the reader. :-)

Hope that helps,

[1nickt]

No, it was I. I went too fast while looking at your use of $last/last; - sorry ! But now he has choices :)

As to the OP assuming that there will always be a match ... yep, better test for that! And what if the log is being written by a daemon that is getting input from multiple threads? I've had logs with entries that weren't chronological, even from apache, and until you parse and sort the entries you can't guarantee they are in order ... so the "flag" approach you and I offered would miss any entries that were separate from the group by a non-matching one ... and so it goes :-)

That's why it's unfortunate that so many of the newbs seem to be interested only in having $it work rather than learning tools to get $it or $unknown_future_thing to work.