Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

why is Camel saying you can safely <> a growing file?

by Mark_Galeck (Novice)
on Jan 04, 2013 at 04:15 UTC ( #1011546=perlquestion: print w/ replies, xml ) Need Help??
Mark_Galeck has asked for the wisdom of the Perl Monks concerning the following question:

Hello, Camel on page 779 in the seek man section says " (...) follow growing file like this (...)
while (<LOG>) { grok($_); } sleep 15; seek LOG, 0, 1;
This contradicts what I am seeing! If the file is growing and I do <LOG>, sometimes I will get a line that does not end in '\n', and then the next call will get the continuation of the line. Clearly the WRONG thing. I would think one needs to check for the end of the line, and if it is not '\n' then seek back to the beginning and then sleep, or something like that. Clearly Camel cannot be wrong. So I must be wrong. Where am I wrong?

Comment on why is Camel saying you can safely <> a growing file?
Download Code
Re: why is Camel saying you can safely <> a growing file?
by Anonymous Monk on Jan 04, 2013 at 04:21 UTC

    Where am I wrong?

    I don't own Camel, even your version, so I can't tell if you're misquoting :)

Re: why is Camel saying you can safely <> a growing file? (buffering)
by tye (Cardinal) on Jan 04, 2013 at 05:07 UTC

    That just means the code writing to the log is doing it badly...

    Appending a line to a log can be done atomically quite simply in Unix. That is the most typical situation and one which would never allow the problems that you are seeing. That is typical because it is very common for more than one process to be writing to a file if it is called a "log file" and not doing that means that you'd end up with intermixed parts of lines.

    The most likely way I would expect what you are seeing is due to the writing to the log being done "fully buffered" such that instead of lines being written to the file, the batches of text that get written to the file are "one buffer full". A more serious problem with that situation is that it can take forever for information to actually make it into the log. So turn off buffering or switch to "line buffered" mode (in the program writing to the log file).

    So writing "fully buffered" to a "log file" is a bad practice for several reasons and so wasn't the case being considered by the author of the advice you read.

    I would think one needs to check for the end of the line, and if it is not '\n' then seek back to the beginning and then sleep, or something like that.

    Well, that's an overly complicated way to deal with that problem. If you don't have a trailing newline, then just <> again and append the result to the previous data:

    my $prev = ''; while( 1 ) { while( <LOG> ) { $_ = $prev . $_; if( ! /\n$/ ) { $prev = $_; next; } $prev = ''; grok($_); } sleep 15; seek LOG, 0, 1; # Clear EOF flag }

    - tye        

Re: why is Camel saying you can safely <> a growing file?
by Mark_Galeck (Novice) on Jan 04, 2013 at 05:29 UTC
    OK, wow it just happens that the turn of the page, is at the bottom of that code, and on the next page, guess what, it says "depending on how standard your standard IO is, you may need something more like this...". Murphy's law
Re: why is Camel saying you can safely <> a growing file?
by flexvault (Parson) on Jan 04, 2013 at 13:03 UTC

    Mark_Galeck,

    The second or third script I wrote in Perl was to "POP before email". It's changed over the years, but is still running on 10+ machines. I could not get one script to run on all *nix versions until I used the 'tell' function to save the location of the last successful read. What the book doesn't mention is that if the log file is smaller than the saved 'tell' location, you should start at the beginning of the file.

    Good Luck...Ed

    "Well done is better than well said." - Benjamin Franklin

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1011546]
Approved by NetWallah
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (14)
As of 2014-12-22 15:38 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (119 votes), past polls