Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Apache and log4perl (high performance logging?)

by zerohero (Monk)
on Feb 10, 2009 at 19:23 UTC ( #742857=perlquestion: print w/ replies, xml ) Need Help??
zerohero has asked for the wisdom of the Perl Monks concerning the following question:

Could someone with experience provide some guidance as to the use of Log4perl w.r.t. Apache2 (and mod_perl, of course)? The basics are pretty obvious, and I know my way around the 3 technologies pretty well, but there are some performance related details I don't have experience with.

For example, I don't understand the performance impacts of (log4perl) logging in a high traffic web application. The idea of each of 32-64 Apache child processes contending for a lock (e.g. syswrite or semaphore via Log::Log4perl::Appender::Synchronized) and then writing to the disk seems to be an issue, I'm just not sure how big an issue it is. Should logging be minimal and then made verbose when one needs to interrogate? Or do people just run at full detail (who cares, no impact)? Some experience would provide illumination as to how important this is. Also knowing simple details like which appender to use based on experience, would be helpful (e.g. File + newsyslog, versus FileRotate).

Also interestingly, the informative Log4perl FAQ by messr. Mike Schilli mentions 30K ops/sec by an "in-memory" appender, which sounds high for something writing to disk. Call me a tad skeptical on this claim (perhaps it is a measurement of the library's efficiency sans disk writing). Mike also mentions newsyslog, an external program to do the log rolling, rather than using FileRotate, I'm guessing to parallelize and possibly do things like compression (again, I lack experience here, so this is just a guess).

I'd appreciate very much any responses from people with real world experience of what works well and what doesn't to help make the right choices for a scalable, maintainable, production system.

Comment on Apache and log4perl (high performance logging?)
Re: Apache and log4perl (high performance logging?)
by andreas1234567 (Vicar) on Feb 11, 2009 at 09:20 UTC
    I have used Log::Log4perl is production with Apache and mod_perl2, although not in very high traffic applications. So what are the performance impacts of (log4perl) logging in a high traffic web application? The answer is, as always, it depends.

    Running in DEBUG mode (full detail) can seriously slow down performance, so you should be careful when setting up your loggers, appenders and levels. Also, very verbose logging can make the logs hard to read, and you can keep less history than if you were less verbose.

    Benchmarking your application with different log4perl settings is important to understand how it affects you in your environment. Perhaps you should experiment with different disk setups, such as having /var/log/ on a separate disk system? Your I/O performance will most likely be one of the most important factors.

    Do also keep an eye on the is_level functions in order to minimize impact of suppressed messages:

    if($logger->is_error()) { $logger->error("Erroneous array: @super_long_array"); }
    --
    No matter how great and destructive your problems may seem now, remember, you've probably only seen the tip of them. [1]
Re: Apache and log4perl (high performance logging?)
by jasonk (Parson) on Feb 11, 2009 at 20:53 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://742857]
Approved by bassplayer
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (13)
As of 2014-09-23 13:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (221 votes), past polls