Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

Greetings wise brothers.

In my day to day perl projects, I mostly use Log4perl for logging. It is flexable and it works quite nicely.

Recently I have joined a complex project involving a cluster of dozens of servers, each running several different daemons. The cluster processes arround 100_000 jobs per week, with different parts of each job running on different servers in the cluster. Log files are stored localy on each server. Each daemon logs to stdout which is redirected to a log file.

Needless to say it is difficult to investigate issues when stuff goes wrong as there are many log files on different servers to look at, and the only way to isolate log messages that relate to a particular job is to use grep. The volume of the log files is also a problem. Currently we log a lot of detail as it might be needed for later anasys, but this makes for huge log files and slow greping of them.

For that reason I am investigating putting the log messages into a database using Log::Log4perl::Appender::DBI or such like, as this would have the advantage of putting all the logs in one place, and make it easier to purge verbose messages on a schedule while keeping errors for much longer.

My problem is how to view and filter the logs in a user freindly way. While in theory I could write some SQL like: Select * from logs where jobID = 12345 in practice it is not a user friendly way of doing things.

I have seen that log4javascript has a web interface tool for log messages that lets you search and filter log messages. (See their demo). Does anyone know of a web interface that can be used to view and filter log messages from a database? My google searches are coming up empty.

In reply to How to view and filter logs in a database by chrestomanci

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and all is quiet...

    How do I use this? | Other CB clients
    Other Users?
    Others wandering the Monastery: (2)
    As of 2018-03-21 23:39 GMT
    Find Nodes?
      Voting Booth?
      When I think of a mole I think of:

      Results (272 votes). Check out past polls.