Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

There is a space issue in the prod server, its require to delete the old log lines i.e. the log lines which tracked on 2011 and past from all the applications event-logs

by perladdict (Chaplain)
on Feb 12, 2013 at 18:35 UTC ( #1018424=perlquestion: print w/ replies, xml ) Need Help??
perladdict has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,
There is a requirement to delete the old log lines from event-logs in production server in order to free the disk space
Below is the Unix directory structure for each application to navigate to the event-logs.
$ cd /rms $ ls 088 533 788 802 916 lib 267 535 789 822 917 lib.old 268 537 790 834 919 LMPROD 270 600 791 841 AXSDBIMG log 272 6000 792 855 CVSROOT lost+found 276 779 794 872 DBG ques 287 781 795 888 DBI scripts 288 783 796 910 DBT scripts.old 406 784 797 911 GENEOS test.tar 450 785 798 912 home transfer 490 786 799 915 KL utils
Above are all the application branch codes except log,lib.old,log,LMPROD,ques,scripts,test,utils,transfer,lib,test.tar
Each of the branches are having different applications event-log,the TMP directory is having all the applications event log for a branch
I need to navigate to each branch's TMP directory in order to view the event-logs
for example: "cd /rms/799/cool/data/TMP" is the path of event-logs for a branch 799.
$ cd /rms/799/cool/data/TMP -rw-rw-r-- 1 coolvas1 coolvas1 4325 Jan 4 15:04 COOL-event-log.L +BOX -rw-rw---- 1 coole3 coole3 297 Jan 25 03:01 COOL-event-log.E +3ASIA -rw-rw-r-- 1 coolbrz coolbrz 20526 Feb 12 05:27 COOL-event-log.B +REEZE -rw-rw---- 1 coolmint coolmint 307297 Feb 12 08:30 COOL-event-log.M +INT
Each application's event-logs for a branch is having application specific group names and user name, in order to view the event logs i need to log-in with application specific usernames.In order to gain the disk space i need to delete the old log line i.e. log lines which is tracked in "/2011 and before
Below is the sample event-log
>coolmint started sh-MINTlink on 24/08/2012 at 08:30:09 INBC file found, Business date fetched is: 20120823 Received Parameters: APP: MINT, BRANCH: 799 and RECORDING DATE: 201208 +23 Validating file: CS-TWTPOUTC.20120823.1 CS-TWTPINBC.20120823.1 CS-TWTP +OREF.20120823.1 CS-TWTPIREF.20120823.1 CS-TWTPNAK.20120823.1 Transfering file: CS-TWTPOUTC.20120823.1 CS-TWTPINBC.20120823.1 CS-TWT +POREF.20120823.1 CS-TWTPIREF.20120823.1 CS-TWTPNAK.20120823.1 > started MINT recording for procdate 20120823 on 2012/08/24 at 08:30: +10 Application Level BACKDATE_DAYS NOT set, hence considering Branch leve +l days a CS-TWTPOUTC.20120823.1 19K a CS-TWTPINBC.20120823.1 54K a CS-TWTPOREF.20120823.1 1K a CS-TWTPIREF.20120823.1 1K a CS-TWTPNAK.20120823.1 1K ============================================= Adding page break to the EOF for CS-TWTPOUTC.20120823.1 Adding page break to the EOF for CS-TWTPINBC.20120823.1 Adding page break to the EOF for CS-TWTPOREF.20120823.1 Adding page break to the EOF for CS-TWTPIREF.20120823.1 Adding page break to the EOF for CS-TWTPNAK.20120823.1 usage: sleep time ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~ > coolmint started sh-MINTlink on 25/08/2012 at 08:30:09 INBC file found, Business date fetched is: 20120824 Received Parameters: APP: MINT, BRANCH: 799 and RECORDING DATE: 201208 +24 Validating file: CS-TWTPINBC.20120824.1 CS-TWTPOUTC.20120824.1 CS-TWTP +OREF.20120824.1 CS-TWTPIREF.20120824.1 CS-TWTPNAK.20120824.1 Transfering file: CS-TWTPINBC.20120824.1 CS-TWTPOUTC.20120824.1 CS-TWT +POREF.20120824.1 CS-TWTPIREF.20120824.1 CS-TWTPNAK.20120824.1 > started MINT recording for procdate 20120824 on 2012/08/25 at 08:30: +10 Application Level BACKDATE_DAYS NOT set, hence considering Branch leve +l days a CS-TWTPINBC.20120824.1 11K a CS-TWTPOUTC.20120824.1 53K a CS-TWTPOREF.20120824.1 1K a CS-TWTPIREF.20120824.1 1K a CS-TWTPNAK.20120824.1 1K ============================================= Adding page break to the EOF for CS-TWTPINBC.20120824.1 Adding page break to the EOF for CS-TWTPOUTC.20120824.1 Adding page break to the EOF for CS-TWTPOREF.20120824.1 Adding page break to the EOF for CS-TWTPIREF.20120824.1 Adding page break to the EOF for CS-TWTPNAK.20120824.1 usage: sleep time ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~ > coolmint started sh-MINTlink on 27/08/2012 at 08:30:09 INBC file found, Business date fetched is: 20120825 /rms/transfer/799/mint/*OUTC.20120825*: No such file or directory Received Parameters: APP: MINT, BRANCH: 799 and RECORDING DATE: 201208 +25 Validating file: CS-TWTPINBC.20120825.1 CS-TWTPIREF.20120825.1 CS-TWTP +NAK.20120825.1 Transfering file: CS-TWTPINBC.20120825.1 CS-TWTPIREF.20120825.1 CS-TWT +PNAK.20120825.1 > started MINT recording for procdate 20120825 on 2012/08/27 at 08:30: +09 Application Level BACKDATE_DAYS NOT set, hence considering Branch leve +l days a CS-TWTPINBC.20120825.1 7K a CS-TWTPIREF.20120825.1 1K a CS-TWTPNAK.20120825.1 1K ============================================= Adding page break to the EOF for CS-TWTPINBC.20120825.1 Adding page break to the EOF for CS-TWTPIREF.20120825.1 Adding page break to the EOF for CS-TWTPNAK.20120825.1 usage: sleep time ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~ > coolmint started sh-MINTlink on 28/08/2012 at 08:30:09 INBC file found, Business date fetched is: 20120827 Received Parameters: APP: MINT, BRANCH: 799 and RECORDING DATE: 201208 +27 Validating file: CS-TWTPOUTC.20120827.1 CS-TWTPINBC.20120827.1 CS-TWTP +OREF.20120827.1 CS-TWTPIREF.20120827.1 CS-TWTPNAK.20120827.1 Transfering file: CS-TWTPOUTC.20120827.1 CS-TWTPINBC.20120827.1 CS-TWT +POREF.20120827.1 CS-TWTPIREF.20120827.1 CS-TWTPNAK.20120827.1 > started MINT recording for procdate 20120827 on 2012/08/28 at 08:30: +09 Application Level BACKDATE_DAYS NOT set, hence considering Branch leve +l days a CS-TWTPOUTC.20120827.1 14K a CS-TWTPINBC.20120827.1 47K a CS-TWTPOREF.20120827.1 1K a CS-TWTPIREF.20120827.1 1K a CS-TWTPNAK.20120827.1 1K Adding page break to the EOF for CS-TWTPOUTC.20120827.1 Adding page break to the EOF for CS-TWTPINBC.20120827.1 Adding page break to the EOF for CS-TWTPOREF.20120827.1 Adding page break to the EOF for CS-TWTPIREF.20120827.1 Adding page break to the EOF for CS-TWTPNAK.20120827.1 ============================================= usage: sleep time ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~ > coolmint started sh-MINTlink on 29/08/2012 at 08:30:08 INBC file found, Business date fetched is: 20120828 Received Parameters: APP: MINT, BRANCH: 799 and RECORDING DATE: 201208 +28 Validating file: CS-TWTPOUTC.20120828.1 CS-TWTPINBC.20120828.1 CS-TWTP +OREF.20120828.1 CS-TWTPIREF.20120828.1 CS-TWTPNAK.20120828.1 Transfering file: CS-TWTPOUTC.20120828.1 CS-TWTPINBC.20120828.1 CS-TWT +POREF.20120828.1 CS-TWTPIREF.20120828.1 CS-TWTPNAK.20120828.1 > started MINT recording for procdate 20120828 on 2012/08/29 at 08:30: +09
From the above log i need delete the log lines below the tracking datelines by keeping the date line as it is in the log. Sorry for the long posting,below is the code i am trying to implement this in perl as well in shell script, but no luck...
#!/usr/bin/perl -w @branches = `ls -ld /rms/???/cool/data/HOLDING | awk -F" " '{print $NF +}' | cut -f3 -d"/" | sort | uniq`; #print "@branches\n"; foreach(@branches) { chomp $_; $var1 = "/rms/"; $var2 = $_; $var3 = "/cool/data/TMP"; @absdirpath = $var1.$var2.$var3; #print "@absdirpath\n"; foreach(@absdirpath) { @eventlogs = `ls $_ |grep "COOL*`; #print "@eventlogs"; foreach (@eventlogs) { @listing = `ls -lrt $_`; print "@listing"; } #@grp = `ls -lrt | cut -d " " -f6`; #print "@grp\n"; foreach $logs (@eventlogs) { $logs =~ s/^\s//; print "$logs\n"; ($logname,$appname) = split(/\./,$logs); @apps = $appname; #print "@apps"; } } }
# To get the branches for branch in `ls -ld /rms/???/cool/data/HOLDING | awk -F" " '{print $ +NF}' | cut -f3 -d"/" | sort | uniq` do # every branch cd /rms/$branch/cool/data/TMP # To get the respective application for app in `ls -ld COOL-event-log.* | awk -F" " '{prin +t $NF}'|cut -f2 -d"."|sort|uniq` do echo "Performing Deletion of lines before 2011 + for: Application : $app, Branch: $branch" echo "======================================== +======================================== " # To get the respective application group for grp in `ls -ltr COOL-event-log.$app|awk '{print $3}'` do newgrp $grp if [ -f COOL-event-log.$app ] then tot_cnt=`wc COOL-event-log.$app|awk ' { print $1 }'` del_cnt=`grep -n '/2011' COOL-event-log.$app|head -1|cut - +f1 -d":"` rem_cnt=`expr $tot_cnt - $del_cnt - 1` tail -$rem_cnt COOL-event-log.$app > COOL-event-log-new.$a +pp rm -f COOL-event-log.$app mv COOL-event-log-new.$app COOL-event-log.$app sudo chown $grp:$grp COOL-event-log.$app else echo no file for interface $app fi done done done
Dear monks, please assist me to get done this, as there is a deadline for me in order to avoid outage.Thanks in-advance.

Comment on There is a space issue in the prod server, its require to delete the old log lines i.e. the log lines which tracked on 2011 and past from all the applications event-logs
Select or Download Code
Re: There is a space issue in the prod server, its require to delete the old log lines i.e. the log lines which tracked on 2011 and past from all the applications event-logs
by Illuminatus (Curate) on Feb 12, 2013 at 23:20 UTC
    First of all, find should give you all of your files with a single command:
    find /rms -name 'COOL-event-log*' -exec <perl-script> {} ';'
    Unless there are files in the /rms directory tree with the same name that you don't want to edit. You should be able to modify the command to exclude those.

    Does this application keep these logfiles open for as long as it's running, and is it running all the time? Because if it doesn't close the file between runs, then you can't do this at all.

    Are you sure the application won't try to run while your script is running? If it does, then you'll have a race-condition.

    Parsing a single event file in a perl script to remove old entries is a pretty simple program. It can open the file passed to it, write the edited results to the '.new' file, then move it over

    fnord

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1018424]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (15)
As of 2014-09-16 19:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (46 votes), past polls