Hi All,
Thanks a lot for so many ideas.
To be more clear, in a single day system generates 5-6 lakh files in just 3-4 directories consuming around 10-14 GB space per day (individual file size is not huge - 12345 bytes, 5-6 digits number in bytes). There are files of 4-5 different patterns but count is huge. No date timestamp in file name (e.g ABC_YYYMMDD.log), just random 8 digits number in file names (i.e. ABC_????????.log)
As per my observation, glob is not consuming high CPU. As soon as we are entering while loop, then can see CPU spikes. After commenting out mv, zip, unlink, just keeping print statements, I noticed CPU around 80%.
After adding below in "next" section, CPU is under control 10-12% or even low to 4% sometime.
select (undef, undef, undef, 0.250);
but already process running slow because of huge number of files and stat calls as highlighted by you "all" above, cant afford sleep.
Once again thanks to all of you, will try out various ideas given by you.
Thanks for letting me know about File::Find::Rule as well.
