Beefy Boxes and Bandwidth Generously Provided by pair Networks DiBona
go ahead... be a heretic
 
PerlMonks  

Re: How to view and filter logs in a database

by Anonymous Monk
on Oct 26, 2012 at 15:28 UTC ( #1001112=note: print w/ replies, xml ) Need Help??


in reply to How to view and filter logs in a database

If the volume of log files is bad, the volume in a database might be even worse of a problem. A scanner, maybe hooked to logrotate, could comb through the files and pick-out the meaningful messages. Then, gzip the files for a couple of days and then deep-six them. The script that gathers up the log messages could put them into whatever database table or tables you want, and the tables in question would be indexed appropriately.


Comment on Re: How to view and filter logs in a database
Re^2: How to view and filter logs in a database
by chrestomanci (Priest) on Oct 29, 2012 at 14:53 UTC

    Fair point about the volume of the data, though I am hoping that logging to a database will help solve that.

    At the moment I am logging quite verbosely, as the verbose logs are needed when working out why something crashed, but I am also keeping the log files for about two months, for statistics and reporting. (Which does not need the vebose messages)

    My plan with logging to a database, is that I can log verbosely, and then purge all the verbose messages after a couple of days (unless something crashes), to leave behind a much smaller volume of non verbose messages, that will be kept for the standard 2 month window.

    This would rely on using a database that will efficently compact and re-use space from deleted records. I have allready had bad experences in that area with MySQL. Postgres or Couch look to be promising.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1001112]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (19)
As of 2014-04-23 09:33 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (541 votes), past polls