http://www.perlmonks.org?node_id=842301

I am a huge fan of Smart::Comments, to the point where it astonishes me that few Monks are. I think it is absolutely the bee's knees. It does have its limitations and it has one particular shortcoming, which I'll discuss briefly at the end of this node.

My basic thesis is that although Smart::Comments is a source filter, this is not an issue, since it is easily and completely disabled in production. It does a difficult job well and I haven't seen an alternative I like.

Demo

Do this:

use Smart::Comments; # Enable special comments for debugging my $data_structure = { a => [ 1, 2, 3 ], }; my $scalar = 42; ### This is a comment. ### <now> I'm happy. ### $data_structure ### $scalar #my $max = 2**4; # slow my $max = 2**2; # fast for (0..$max) { ### for1... done sleep(1); }; for (0..$max) { ### for2 |===[%] | sleep(1); };

The first smart comment is printed literally. The second is timestamped. The third prints the name of $data_structure and dumps its value; the fourth does the same for a simple $scalar. The fifth and sixth smart comments demo progress bars. (You might wish to play around with $max. The trouble with demoing progress bars is that, well, you only see them if you are forced to wait.)

Smart::Comments does more than this but I think the dumping feature alone is worth the price. Others don't agree and this is my defense.

My View of the Task

I'm a "print statement debugger"; I've used interactive debuggers and don't much like them. In general, I'm wary of anything that introduces another level of meta-operation. When I run my stuff, I just run it; I make executable scripts. I don't run perl with command line options. I like to hope that my code is well enough structured that its bugs aren't too deep.

Actually writing print-for-debug statements is a chore -- a minor one but this is not a place where complexity is desired. I frequently want to dump something, somewhere; so I don't want to type one more character of code than necessary. I'm debugging something in my target code; the last thing I want is a buggy print-for-debug statement. A small simplification is a big win here.

After a short time, my code is littered with such statements, which must all be cleaned up for production. Of course, I often just comment them out, in case I want them again later. The cleanup process is liable to short-between-the-headphones error, as I either fail to comment out or remove all the debugging statements; or I make the contrary mistake, destroying production code.

I don't use the other kinds of smart comments much. I may in future. But I'll defend S::C on its dumping.

To dump a scalar with a regular print statement:

    print '$var: ', $var, "\n";    # DEBUG

Dumping arrays, hashes, and more complex structures is much worse. I have old code with entire dumping subroutines (using Data::Dumper, of course); then the calling syntax is something like:

    _dump( '%var', \%var );        # DEBUG

Leave off a backslash and I'm toast. I'd rather debug my target code than my debugging code. I don't even like print, really; the four keystrokes needed to append a newline annoys me, especially since two are shifted. I pretty much insist on say in all my code. This fails, of course, when for whatever reason, feature 'say' is not available.

But the real gorilla, for me, is twice typing the name of the thing to be dumped. If I do this in a regular print/say statement, then I'm likely to leave off some niceties. If I'm in a hurry, I tend to (in increasing disorder):

1: say "var: $var"; # DEBUG 2: say '$var', $var; # DEBUG 2: say $var; # DEBUG 3: say $var;

In line 1, I've left off the sigil from the label so it doesn't interpolate, which is fine unless there's also a @var in scope. In line 2, I've omitted any delimiter between the label and the value, which may lead to confusion. In line 3, I'm impatient enough not to label the dump at all. In line 4, I'm so concentrated on the underlying task that I omit the flag that lets me search for my litter later on cleanup. I've done all these bad things.

The probability of writing a funked-up print-for-debug statement is a function, with positive correlation, of the code length and number of variables to be dumped. Consider:

    print '$very_long_identifier: ', $very_long_identifier, "\n";    # DEBUG

This is still only a dump of a simple scalar! Complex structures really do require resort to Data::Dumper.

A Solution?

I tried for some time to roll my own dump-for-debug subroutine, which preferably I would wrap in a module and call this way:

    mydump( $var );

Presumably, this bit of vaporware would also accept a simple flag to silence such dumps in production. Of course, the calling overhead would still be there. Or I could manually go through and comment out the little guys.

Another hangup is getting the label, based on the variable name as written in the code. Pad::Walker provides a partial solution. But between lexicals and package variables, this and that, I hit a wall in that direction. The plain fact is that there is only one lexical scope that properly knows what every variable in that scope is named: any called routine, even in the same package, is going to have to break-and-enter to do its dirty work.

I don't say that a Saint couldn't deal with all these issues. I do say I haven't seen it done yet.

Enter Smart::Comments

Since you need to be in scope to get easily the right names for everything in that scope, a source filter is the natural approach. The 'statement' is then:

    ### $var

I don't see any way to golf a char off that. It expands into a call to Smart::Comments::_Dump, which does the dirty work -- and does it very cleanly. Package variables, lexicals, arrays, hashes, and nested structures are all dealt with reasonably. You can try to dump stuff that won't; coderefs don't play well. But mostly, you get sensible output. At worst, caller needs to set up a temporary variable to dump:

my $tmp = scalar @foobar; ### Elements count: $tmp

Another useful feature is that smart comments need not always be introduced with exactly three octothorpes:

use Smart::Comments '###', '####'; ### $foo # $foo will be dumped #### $foo # $foo will be dumped ##### $foo # nothing happens

Any given smart comment can be disabled selectively by prepending another octothorpe. You can re-enable them as a group by adding another string to the use line.

There's still another disabling method available:

use Smart::Comments; ### $foo # $foo will be dumped no Smart::Comments; ### $foo # nothing happens

Objections Answered

It's a source filter.
Yes. I don't use any other source filters for any purpose, so there's nothing for Smart::Comments to conflict with head-to-head. I don't see why a source filter is an outright deal-killer. Like any tool, it can be misused. I need to see a more concrete objection.

It's a source filter, so it should never be used in production.
Agreed. Release production code after it's been debugged. You comment out the use Smart::Comments line itself and all the formerly smart comments become dumb comments, which can of course be re-enabled next development cycle. If you object to too many dumb comments littering the code, it's easy to find and delete them; if you miss one, no matter.

It's a TheDamian module.
Is that merely an ad hominem attack? Or is his track record so bad that one presumes his stuff will eventually blow up?

I have a better way of doing this.
I don't. Please show me.

The Big Fat Shortcoming

Smart::Comments does have a shortcoming which I find serious. This is, it prints all its output to the screen via STDERR. This is hardwired; there's no way to change it, except perhaps by redirecting STDERR elsewhere.

This conflicts with my current testing framework, which captures both STDOUT and STDERR using IO::Capture. I want my debugging output to go to a disk file, so I can pore over it at my leisure, while output from the module under test be undisturbed, so that it tests correctly.

I'm working on a fix. Please hold any comments on the fix to another node. Thank you.

- the lyf so short, the craft so long to lerne -

Replies are listed 'Best First'.
Re: In Defense of Smart::Comments
by BrowserUk (Patriarch) on May 30, 2010 at 23:29 UTC

    I've recommended Smart::Comments here many times, but I'll admit to not using it much recently. What you call "The Big Fat Shortcoming" has never been a problem for me; if I need to retain the output, a simple script 2>log has always done the job perfectly. With the advantage over internally specifying a file, that I can do script 2>&1 | tee log | more which I find very useful.

    The big limitation of S::C for me is its verbosity. I don't need 4 lines of output for every traced variable; one is sufficient. The way it dumps structures is equally pointlessly verbose. And the cutesy for loop tracing wears thin very quickly. I did start to look at substituting Data::Dump for structures and so on, but them something else caught my attention more and I never completed the exercise.

    Like you, I think that the idea of the module, including the source filter, perfectly sound for the purpose, it's just the details that let it down in my view.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: In Defense of Smart::Comments
by GrandFather (Saint) on May 31, 2010 at 01:55 UTC

    I find an interactive debugging environment so much more powerful than what amounts to post mortem analysis of trace information that I've never used Smart::Comments and seldom use Data::Dumper or Data::Dump::Streamer.

    Smart::Comments may be all sorts of wonderful, but with just a little bit more effort learning to use a debugger provides a much better pay off. Of course using a good IDE (with integrated debugging) boosts productivity by even more.</flamebait>

    True laziness is hard work

      An interactive debug is quite handy. I did manage to use the built in debugger at one point, but I've forgotten how by now due to disuse.

      Since my project is interactive itself, I've built debugging options right in. (Only for the user logged in as root, however)

      • Root can set DEBUG_XYZ preferences to turn on and off debug printing by category (Not just a scalar debug level, but targeted to specific areas of the code).
      • Root can also call an eval command, which gives wildcard power to hunt down problems, inject tests and right wrongs while the app is still running.

      Aside from the prerequisite of interactivity, my app also has a giant hash tree containing all of the important state info. It is used to save and restore to/from disk between runs, but it also provides straightforward access to almost everything for debugging.

      As far as leaving the debug in goes, a bunch of print Blah if DEBUG; aren't going to hurt. Knocking exponents off the Big-O and filtering down the value of N is where the real speed is found.


      For non-interactive apps using print-foo-if-debug-constant is what I used to do. The benefit of category-based debugging, with the compile time removal of that code when the constants are set to false.
      EG: print "Garthok was Gnarfled with a $weapon\n" if DEBUG_GARTHOK || DEBUG_COMBAT;
      Smart Comments can't do that with octothorpe counts.
      I have thought about mixing in Smart Comments for the looping features, but I never really need to look at loops themselves, only a problem inside a particular iteration of the loop.

        There is nothing (much) magic about CGI scripts. Just run them as though from the command line. The IDE I'm using simulates a CGI environment which helps, but isn't essential.

        There is no need to debug on the remote machine either of course. In fact it's probably safer to use an off line copy of databases and anything else required by the script when debugging and do the lot on your local machine.

        The built in debugger is text mode so you can use that without trouble on your live machine over a plain-text ssh connection should you decide that sort of chain saw juggling is for you.

        True laziness is hard work
Re: In Defense of Smart::Comments
by toolic (Bishop) on May 31, 2010 at 01:11 UTC
    I am a huge fan of Smart::Comments, to the point where it astonishes me that few Monks are
    Maybe most monks are unaware of its existence. While I applaud and appreciate your testimonial, I am surprised you feel the strong need to defend the module. As a daily participant here at the Monastery for the last 2+ years, I am unaware of a negative concensus opinion of the module.

    Everyone who's ever written a line of Perl code knows about the built-in print function. Every experienced Perl hacker knows about the core Data::Dumper module. First, you have to find out about Smart::Comments. Then you must download/install the CPAN module -- a minor, but real, barrier to usage for some people.

    It's a TheDamian module. Is that merely an ad hominem attack? Or is his track record so bad that one presumes his stuff will eventually blow up?
    The author offers advice on usage of his modules: Categorized Damian Modules

      Perhaps I can hope to have lowered the barrier?

      - the lyf so short, the craft so long to lerne -
Re: In Defense of Smart::Comments
by Your Mother (Archbishop) on May 31, 2010 at 02:39 UTC
    Is that merely an ad hominem attack? Or is his track record so bad that one presumes his stuff will eventually blow up?

    This strikes me as pretty strange. Apart from TimToady, merlyn, and a handful of others like audreyt there is none more respected and admired than TheDamian.

    Smart::Comments is something I've looked at now and then and frankly your example shows me why I'm glad I never sunk any time into hacking with it. It is completely unclear what the sample does just from looking at it. I can read any Test::More and Log::Log4perl or even Test::Class -- or even Test::Inline for that matter -- code and understand it immediately without consulting the docs. I also personally dislike any formatting constraints to make code work ### Even that.

    I'm not coming out against Smart::Comments. If you like, great! Just saying I think it's a perfectly reasonable thing to not choose and it seems a rather roundabout way of replacing other tools which are in wider use and translate across languages, frameworks, and platforms much more transparently.

    (update: fixed log4perl link.)

      It strikes me as pretty strange, too. Yet this is a common CB reaction to mention of S::C. I've read it enough times; I felt it required an answer.

      Personally, I don't find DC short on hubris. I admire the bravery of a man who says flatly, in print on paper, "Use 78-column lines." I've been in places in Cupertino where that would start a fight involving thrown bar stools. I like him even better when he backs up his flat imperatives with pages of solid reasoning.

      DC takes risks; he pushes Perl to the edge. Sometimes he wins; when he does, he wins big. Occasionally, he flops, to scale. I don't see that as a problem but perhaps some do.

      Certainly, I don't say everyone must use S::C. Here, I'm not even recommending it (much). It works for me (to a point); it works well enough that I'm now investing effort improving/extending it. But this node merely says, "Smart::Comments is not evil."

      - the lyf so short, the craft so long to lerne -
      there is none more respected and admired than TheDamian.

      As much as he is admired, his modules are his toys, they're under documented and under maintained. Once he is bored with them, because he is so clever, the bugs in his modules can be very difficult to fix.

        the bugs in his modules can be very difficult to fix.
        Not true. It can be very difficult to have him patch and release an update of his module, but he has no problems letting others take over maintainance of his modules.

        Go ahead, volunteer to maintain Smart::Comments and apply any fix you want.

Re: In Defense of Smart::Comments
by waba (Monk) on Jun 01, 2010 at 05:43 UTC

    I've used Smart::Comments a bit after reading Damian's PBP, but I find Log::Log4perl much more useful overall:

    • In :easy mode, writing DEBUG calls is just as simple as writing ### comments.
    • The logging level can be reconfigured through command-line options or a configuration file (this was my main gripe with S::C).
    • I can naturally grow my usage to a full-featured logging framework.
    • It's easier to "sell" to your boss/customer, should you ever need to enable tracing/debug outside of your workstation.
      Half of point #2 is not valid as you can do the same with Smart::Comments.
      perl ... off perl -MSmart::Comments ... on (###) perl -MSmart::Comments,'###' ... on (###) perl -MSmart::Comments,'###','####' ... on (###, ####)

        I meant script command-line options, as in ./myscript -vvv. Users do not know (nor they should!) about perl and its option syntax to provide a tracelog.

        So yes, it's technically possible, but even when debugging my own script, typing perl -MSmart::Comments,'###','####' /usr/local/bin/myscript is not really convenient.

        Half of point #2 is not valid as you can do the same with Smart::Comments.
        Additionally, Log::Log4perl allows you to modify the verbosity of a running application (using the init_and_watch flag, albeit with a small overhead). This can be very handy if a production system shows symptoms one wants to investigate while policy prevents restart.
        --
        No matter how great and destructive your problems may seem now, remember, you've probably only seen the tip of them. [1]
Re: In Defense of Smart::Comments
by andreas1234567 (Vicar) on Jun 06, 2010 at 14:06 UTC
    Smart::Comments does have a shortcoming which I find serious. This is, it prints all its output to the screen via STDERR. This is hardwired; there's no way to change it, except perhaps by redirecting STDERR elsewhere.
    This is just one on many reasons to use Log::Log4perl instead of Smart::Comments. I often use it together with Log::Dispatch::FileRotate to retain a given number of logs of a known size to be able to backtrack problems after the fact.
    --
    No matter how great and destructive your problems may seem now, remember, you've probably only seen the tip of them. [1]
      just a remark I like smart comments But from the past i documented my scripts (sh,perl,c,...) with many lines of form: ### @(#)description of prog, author, dependencies, ... Obviously there could arise problems if we use this style together with smart comments NB the '### @(#)' style documentation goes back to the (not more used?) "what" programm, part of the (not more used?) "documenters work bench" :-) Simple solution is "use Smart::Comments '####';"
Re: In Defense of Smart::Comments
by {}think (Sexton) on Jun 23, 2011 at 12:56 UTC

    I'm somewhat enamored of the Smart::Comments module, but have found several shorcomings not mentioned above:

    1. It stops Perl's debugger from performing Run (R) commands for some reason. I have not researched this at all, but have definitely seen that (R) will run your code until the next breakpoint or the next Smart::Comment (sigh).
    2. You can print VARB or LABEL:VARB, but you can't print arbirary strings containing VARBs, such as
                      ### $0 starting
                      ### $var of $var
                      ### $var $var
      
    3. YOU CANNOT have a colon in your Smart comments - the following would cause compile to fail!
                      ## this is record $var: all is well
    4. In-line SMARTCOMMENTS (meaning those to the right of code) do not seem to work. They only work on loops progress bars.
                      $var++;  ### $var
                      ### Did you get that? No, you didn't.
      

    {}think; #Think outside of the brackets

      I've been working on the derivative Devel::Comments.

      1. S::C inserts a breakpoint at every smart comment. My thought was Why. Some developers are interactive debugger guys, some are print-statement debugger guys. If you want both, well, Conway hardcoded in these breakpoints. If you file a (feature request) bug against D::C, I'll make breakpoint insertion optional in 2.0.0.

      2. S::C strongly prefers you to use labelled expressions; I agree it's an issue since I think its best feature is that it generates labels for you. But it's going to have a hard time generating labels for expressions.

      Here's what works in your use cases:

      my $var = 'foo'; ### "$0 starting" ### $var of $var: "$var of $var" ### $var $var: "$var $var"

      3. Don't forget that ## is not a valid introducer. You must use at least ###.

      In any case, yes, the colon is special in smart comments, signifying the end of the label. You don't want to put in two of them, either. The monster regexes that do the source filtering do not like this. You'll notice that a user who filed a bug against an unrelated issue also objects to privileged colons: https://rt.cpan.org/Ticket/Display.html?id=69712

      4. There are excellent reasons to forbid trailing smart comments. To be certain a sequence of octothorpes was intended as a smart comment introducer (and not just some literal text inside something else) would require D::C to parse the whole source code. That said, I'd like trailing smart comments, too. Twist my arm.

      As far as the loop progress bars, they are messy and perhaps not of interest to serious developers. If I don't hear strong support, I'm going to drop them in D::C 2.0.0; I may replace them with more abstract loop-tracing.

      Feste: Misprison in the highest degree. Lady, cucullus non facit monachum. That's as much to say as, I wear not motley in my brain....