ack has asked for the wisdom of the Perl Monks concerning the following question:
Over the past few years, I've been progressively interested in some of the more sophisticated and useful strategies to integrate logging into my Perl scripts. My interest is both for debugging and maintenance purposes and also for capturing the results of my scripts.
But what I have found is that as I integrate logging into my scripts, it is becoming harder and harder to see and follow the logic of the "heavy-lifting" that my scripts are accomplishing...in essence, I am having trouble "seeing the forest for the trees."
My scripts are not continuous-running or long-running scripts; though they often taken many minutes to complete. Many of them are a multitude of scripts that are integrated together using some master control script that doles out the work to the other scripts.
I used to "roll my own" logging so that I could print results to the screen and to log files...but it was tedious and time-consuming to reinvent the wheel every time I wrote a script. So I have, in the last few months, migrated to Log4Perl. It is wonderful and gives me everything I need and a tremendous amount of flexibility...just like it promised to do. I am still a novice and am learing more and more every day about how to use it more effectively.
But with its power comes in vivid colorful obviousness the old problem that I found even with my own home-grown logging: the plethora of logging invocations throughout my code makes it increasingly difficult to follow the real logic of my scripts. The logging invocations are distracting and become "noise" in my logic.
This seems like something is wrong and seems to almost defeat the whole idea of flexible, integrated and pervasive logging in scripts.
So my interest is: How does one balance the effective use of logging while maintaining the sense and view of the intrinsic logic of the scripts being logged? How does one ensure that the logging doesn't become so much "noise" that the original intent of the code is lost?
UPDATE: Thanks to everyone for all the great advice and comments. One thing I didn't note (slipped my mind) is that most of the logging that I use is really not for debugging. Rather I use it to allow me a lot of flexibility in how to capture the script's results. Using, for example, Log4perl I can easily have the output go just to the screen...which I use for most initial debugging and output formatting adjustments, etc...or with simple changes I can have the output got to a log file(s) and to the screen. I can also get only certain types of output to the screen (for example, I can just send notifications when certains stages are processing completed to the screen and a log file and then have all of the details go just to the log file). Even with Log4perl I still tend to do, as one responder apparently does, too...loggings or prints that I ultimately remove when the application script gets into production. That sort of helps me "clean up" the script to regain a sense of the logic and script flow. Anyway, thanks to all of the great feedback. It is comforting to know that I'm not the only that's faced the dilema of balancing code and logging readability.