Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re: why is Camel saying you can safely <> a growing file? (buffering)

by tye (Sage)
on Jan 04, 2013 at 05:07 UTC ( #1011551=note: print w/replies, xml ) Need Help??


in reply to why is Camel saying you can safely <> a growing file?

That just means the code writing to the log is doing it badly...

Appending a line to a log can be done atomically quite simply in Unix. That is the most typical situation and one which would never allow the problems that you are seeing. That is typical because it is very common for more than one process to be writing to a file if it is called a "log file" and not doing that means that you'd end up with intermixed parts of lines.

The most likely way I would expect what you are seeing is due to the writing to the log being done "fully buffered" such that instead of lines being written to the file, the batches of text that get written to the file are "one buffer full". A more serious problem with that situation is that it can take forever for information to actually make it into the log. So turn off buffering or switch to "line buffered" mode (in the program writing to the log file).

So writing "fully buffered" to a "log file" is a bad practice for several reasons and so wasn't the case being considered by the author of the advice you read.

I would think one needs to check for the end of the line, and if it is not '\n' then seek back to the beginning and then sleep, or something like that.

Well, that's an overly complicated way to deal with that problem. If you don't have a trailing newline, then just <> again and append the result to the previous data:

my $prev = ''; while( 1 ) { while( <LOG> ) { $_ = $prev . $_; if( ! /\n$/ ) { $prev = $_; next; } $prev = ''; grok($_); } sleep 15; seek LOG, 0, 1; # Clear EOF flag }

- tye        

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1011551]
help
Chatterbox?
[choroba]: Morning!
[Corion]: Hi choroba, Discipulus!
[Corion]: Yesterday I've been mulling over how to best generate HTTP requests from permutations of values but I haven't found a nice API for passing in the "template" of the HTTP request yet. I guess I'll have to do a SoPW for that
[Corion]: The API itself will basically be my $iter = generate_http_requ ests(method => 'GET', url => '/settings/:name', headers => ???, get_params => ['foo','bar']), but I'm not sure how to parametrize values in the headers and how to specify lists of ...
[choroba]: On the other hand, lots of options to receive the requests :-)
[Corion]: ... values to be used. For example, I think for headers, one would want to have various kinds of Content-Encoding headers, but for the get_parameters one would have various kinds of Bobby Tables
[choroba]: What about [metadoc:// Algorithm::Loops]?
[Corion]: choroba: Yeah, but handing off the request to Dancer,Plack, Mojolicious,LWP is easy once I have the data filled into some structure ;))
[choroba]: Algorithm::Loops
[Corion]: choroba: I'm using that to generate the permutations, but I don't know how the user can pass the intended values to my function in a sane way

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (5)
As of 2017-01-17 08:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you watch meteor showers?




    Results (152 votes). Check out past polls.