http://www.perlmonks.org?node_id=1140867

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

hello,

i use different appenders, with different thresholds. i like to aggregate or combine log entries by time or by number of log entries. i played with Log::Log4perl::Appender::Buffer but i had no luck. it buffered the wrong logger or didn't buffered at all or log entries disappeared or got doubled. simply: a mess.

the only thing i like to achive:

simply combine multiple log entries into one, big chunk. it's ok that all messages share a timestamp, i don't need a line number or something. just log, what happens, not when and where.

you might say: concatenate the messages in a string or something like that. but thats off what i need. i don't like to handle a string beside.

do you know a solution or do i need to write my own composite appender? (which is not my top prio, .= would be easier alternative)

thank you for your time.

Replies are listed 'Best First'.
Re: combine logentries with Log::log4perl
by shadowsong (Pilgrim) on Sep 03, 2015 at 10:22 UTC

    Could you provide us with a bit more information?

    simply combine multiple log entries into one, big chunk. it's ok that all messages share a timestamp, i don't need a line number or something. just log, what happens, not when and where.

    Sounds like Log::Log4perl::Appender::Buffer ought to be able to accomplish that.. Perhaps you could post the code of your attempt to use it?

      sure, that's just a simplification but it shows
      the same strange symptoms:

      use Log::Log4perl qw(:easy); $conf = <<EOT; log4perl.rootLogger = DEBUG, Log1 log4perl.appender.Log1 = Log::Log4perl::Appender::File log4perl.appender.Log1.filename = /tmp/test.log log4perl.appender.Log1.mode = append log4perl.appender.Log1.layout = Log::Log4perl::Layout::SimpleLayout log4perl.appender.Log1.Threshold = INFO log4perl.logger = TRACE, Log2 log4perl.appender.Log2 = Log::Log4perl::Appender::ScreenColoredLevels log4perl.appender.Log2.layout = Log::Log4perl::Layout::PatternLayout log4perl.appender.Log2.layout.ConversionPattern = %m%n log4perl.category = DEBUG, Buffer log4perl.appender.Buffer = Log::Log4perl::Appender::Buffer log4perl.appender.Buffer.appender = Log1 log4perl.appender.Buffer.trigger_level = ERROR EOT Log::Log4perl->init(\$conf); TRACE("message #1 (screen only)"); INFO("message #2"); INFO("message #3"); sleep(10); ERROR("message #4");

      when i run this, i get on screen:

      message #2 message #3 message #4

      and in the file:

      INFO - message #2 INFO - message #3 INFO - message #2 INFO - message #3 ERROR - message #4 ERROR - message #4

      the first two lines of the log file appear at the same time like the ones on the screen. the first line (message #1) to the screen appender just disappears but in the log file message #2, #3 & #4 get doubled when the ERROR trigger got hit.

      very odd behaviour. didn't match my expectations.

      greetings

        You appear to be attempting to log to multiple places, hence all the double ups (and probably the cause of stuff not appearing on screen as expected). Try this:

        use strict; use warnings; use Log::Log4perl qw(:easy); my $conf = <<EOT; log4perl.rootLogger = DEBUG, Log1 log4perl.appender.Log1 = Log::Log4perl::Appender::File log4perl.appender.Log1.filename = /tmp/test.log log4perl.appender.Log1.mode = append log4perl.appender.Log1.layout = Log::Log4perl::Layout::SimpleLayout log4perl.appender.Log1.Threshold = INFO log4perl.logger = TRACE, Log2 log4perl.appender.Log2 = Log::Log4perl::Appender::ScreenColoredLevels log4perl.appender.Log2.layout = Log::Log4perl::Layout::PatternLayout log4perl.appender.Log2.layout.ConversionPattern = %m%n log4perl.category = TRACE, Buffer log4perl.appender.Buffer = Log::Log4perl::Appender::Buffer log4perl.appender.Buffer.appender = Log1 log4perl.appender.Buffer.trigger_level = ERROR EOT Log::Log4perl->init(\$conf); TRACE("message #1 (screen only)"); INFO("message #2"); INFO("message #3"); sleep(10); ERROR("message #4");

        EDIT: You've set the buffer to intercept all messages and only allow DEBUG and higher through. Thats why your onscreen TRACE message disappears. Still looking into the doubling up thing.

        Oh, and I've updated my source - This gets the screen to display correctly, but everything is still doubled in the file.

        Change this line

        log4perl.rootLogger = DEBUG, Log1

        To this:

        log4perl.rootLogger = DEBUG, Buffer

        And... you can remove this line if you'd like the TRACE category to show up on STDOUT

        log4perl.category = DEBUG, Buffer

        Let us know if that matches your expectations? Although deep down I think you'll find you may have to write a custom composite function if you'd like to get a deeper level of customization..

Re: combine logentries with Log::log4perl
by SimonPratt (Friar) on Sep 03, 2015 at 13:30 UTC

    Here:

    use strict; use warnings; use Log::Log4perl qw(:easy); my $conf = <<EOT; log4perl.category = TRACE, Screen # Regular Screen Appender log4perl.appender.Screen = Log::Log4perl::Appender::Screen log4perl.appender.Screen.stdout = 1 log4perl.appender.Screen.layout = PatternLayout log4perl.appender.Screen.Threshold = TRACE log4perl.appender.Screen.layout.ConversionPattern = %m%n # File appender log4perl.rootLogger = DEBUG, Log1 log4perl.appender.Log1 = Log::Log4perl::Appender::File log4perl.appender.Log1.mode = append log4perl.appender.Log1.autoflush = 0 log4perl.appender.Log1.layout = Log::Log4perl::Layout::SimpleLayout log4perl.appender.Log1.filename = test.log log4perl.appender.Log1.Threshold = INFO EOT Log::Log4perl->init(\$conf); TRACE("message #1 (screen only)"); INFO("message #2"); INFO("message #3"); sleep(10); ERROR("message #4");

    In my play around, I found that buffering only reliably works when buffering to screen (which makes sense, given that switching off autoflush effectively buffers file output).

Re: combine logentries with Log::log4perl
by CountZero (Bishop) on Sep 04, 2015 at 08:21 UTC
    Certainly not directly answering your question, but I always wondered why most logging systems employ the rather primitive practice of appending a line to a text file, rather than writing the info to a database file which then could be read, aggregated, compared, selected, extracted, etc. in many different forms with some simple and standard SQL.

    Somewhat more useful for your question: I think it is a bad idea to aggregate or combine log entries at the "logging" stage already. Perhaps you don't need now to know when and where something happened. But are you certain this information may not be needed next week/month/year by you or a colleague? "Knowledge is Power" (guard it well)! Far better -IMHO- to collect as much info as possible and then extract and rearrange into a report as per your needs at that moment.

    CountZero

    A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

    My blog: Imperial Deltronics
      but I always wondered why most logging systems employ the rather primitive practice of appending a line to a text file, rather than writing the info to a database

      Reliability. Appending is probably the safest way to write to a file. In fact, some file systems have a journaling feature - when enabled, block writes are first appended to an on-disk journal, then are written to their actual destination in the disk. Block write status is tracked in memory and fully written blocks are "shift"-ed off the head of the journal whenever the block at the head is fully written.

      Also, having log files being (somewhat) human readable means fewer specialized tools are needed to analyze the logs when something does go very wrong.

        I think appending and even general writing to a database is quite reliable as well. The changes made to the databse files on disk probably go through the same or similar journallng routines as any other disk write.

        Personally I think this is just tradition nobody cared enough about to change yet.

        Actually, this may be one place where Windows is actually more modern than Linux c.s. Windows has a central repository where all kinds of system messages, erors, warnings, ... are kept, not only OS messages, but apps can write to that repository too and there is a simple admin tool ("Event viewer") that allows easy access to this information.

        CountZero

        A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

        My blog: Imperial Deltronics