Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re: How to view and filter logs in a database

by Anonymous Monk
on Oct 26, 2012 at 15:28 UTC ( #1001112=note: print w/replies, xml ) Need Help??


in reply to How to view and filter logs in a database

If the volume of log files is bad, the volume in a database might be even worse of a problem. A scanner, maybe hooked to logrotate, could comb through the files and pick-out the meaningful messages. Then, gzip the files for a couple of days and then deep-six them. The script that gathers up the log messages could put them into whatever database table or tables you want, and the tables in question would be indexed appropriately.
  • Comment on Re: How to view and filter logs in a database

Replies are listed 'Best First'.
Re^2: How to view and filter logs in a database
by chrestomanci (Priest) on Oct 29, 2012 at 14:53 UTC

    Fair point about the volume of the data, though I am hoping that logging to a database will help solve that.

    At the moment I am logging quite verbosely, as the verbose logs are needed when working out why something crashed, but I am also keeping the log files for about two months, for statistics and reporting. (Which does not need the vebose messages)

    My plan with logging to a database, is that I can log verbosely, and then purge all the verbose messages after a couple of days (unless something crashes), to leave behind a much smaller volume of non verbose messages, that will be kept for the standard 2 month window.

    This would rely on using a database that will efficently compact and re-use space from deleted records. I have allready had bad experences in that area with MySQL. Postgres or Couch look to be promising.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1001112]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (6)
As of 2016-09-29 08:11 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Extraterrestrials haven't visited the Earth yet because:







    Results (550 votes). Check out past polls.