Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

(How) Do you document/test your private subroutines?

by stevieb (Abbot)
on Nov 06, 2018 at 22:56 UTC ( #1225330=perlquestion: print w/replies, xml ) Need Help??
stevieb has asked for the wisdom of the Perl Monks concerning the following question:

Recently, a question on testing came up, and it made me think a little bit.

I don't officially document my private methods/functions anywhere in POD or otherwise. Most of my private subs are very small, typically three to 20 lines of code, and are significantly more focus-built than the calls they serve, so I typically just put a comment at the top of the definition (comments in my code are extremely sparse otherwise):

sub _attach_build_log { # attach the cpanm logs to the PASS/FAIL logs sub _copy_logs { # copy the log files out of the temp dir sub _dzil_shim { # shim for working on Dist::Zilla modules sub _exec { # perform the actual *brew build commands (called by test())

For testing, it would be an unusual practice for me to have tests dedicated directly at a private sub, but periodically I find it is useful, and if I change internals, I just let test breakage tell me:

... my $run = $des->_run_end(); is $run, undef, "_run_end() is undef when called out of context"; $run = $des->_run_end(1); is $run, 1, "_run_end() sets itself to true properly"; $run = $des->_run_end(0); is $run, 0, "_run_end() sets itself to false properly";

Curious as to what others do to document core private subs (if at all), and if they take any special procedures to test some of their private subs directly, when testing them through the main API isn't feasible/possible.

Replies are listed 'Best First'.
Re: (How) Do you document/test your private subroutines?
by hippo (Canon) on Nov 06, 2018 at 23:38 UTC

    I don't document private subs in POD because users read POD and private subs are not for users. They can be for devs, however, so I tend to document them where devs would look, ie. in comments in the source. As most private subs are pretty short this approach has few snags, IME.

    To test or not to test depends on the purpose of the sub and the logic therein. If the public subs can be tested simply and give full coverage of the private subs then there's no real need for anything further. The more awkward/gnarly/horrendous ones do receive more special attention however - it can be helpful during dev and thereafter for regressions of course.

    TBH, the work done to polish private subs like this is generally way down the list of priorities. It's not as if there are hard and fast rules which must be adhered to. Let's just say such documentation and tests are aspirations.

      Thanks hippo,

      I agree that the 'rules' are fast and far between for private subs. My private subs are, well, I wouldn't go as far as to say completely 'eloquent', but are typically self-documenting, especially for myself or any collaborators six months down-the-road.

      That said, sometime's I'll add a specific section of documentation in POD (or otherwise) that doesn't exactly list/describe the sub in question, but does include developer-exclusive information in documentation that if they get deep enough, will find and be able to understand what private subs are required by higher-level ones, and easily (hopefully) figure out how they hang together.

Re: (How) Do you document/test your private subroutines?
by tobyink (Abbot) on Nov 07, 2018 at 09:52 UTC

    Because Perl doesn't have true method privacy, Perl developers can mean more than one thing when they talk about a "private" sub/method.

    Sometimes they mean a sub which is not to be used at all outside the module it was defined in. In these cases, I'd suggest you should at least consider making it into a coderef instead of a named sub, so that it cannot be used elsewhere (PadWalker trickery notwithstanding).

    my $run_end = sub { my $self = shift; ...; }; sub some_public_method { my $self = shift; ...; $self->$run_end(); }

    Unless it's really obvious what it does, document its purpose in comments. The question of whether you should test it or not becomes moot — it cannot be used outside the module, so your test scripts can't directly touch it anyway. Any problems which affect the rest of the module should be exposed when you test the public API. Any problems which don't affect the rest of the module probably don't matter.

    Other times, there are functions you have in one class/module that are not intended to be called by end users, but might be called by other classes/modules within your distribution, or overridden by subclasses within your distribution. This is racism. Don't do this. Well, okay, not racism, but that got your attention. It's still discriminating against other modules/classes because they weren't written by you. If it's a function/method which you have found useful/necessary, then other people might find it useful/necessary too, so make it part of your public API. Okay, so maybe it's pretty obscure and 99% of end users won't need it, and you don't want it cluttering up your documentation, but in that case, mention it in a separate "ADVANCED API" section of the documentation (perhaps even in a separate pod file) or document it fully in the comments, making it clear that it's a supported albeit obscure part of your API.

    See also: the middle part of my presentation at the 2014 London Perl Workshop (and there's a cringy video of it on YouTube too).

      Because Perl doesn't have true method privacy, Perl developers can mean more than one thing when they talk about a "private" sub/method.

      Sometimes they mean a sub which is not to be used at all outside the module it was defined in. In these cases, I'd suggest you should at least consider making it into a coderef instead of a named sub, so that it cannot be used elsewhere (PadWalker trickery notwithstanding).

      That's one way, but I prefer the "contract instead of shotgun" way explained in perlmodlib:

      Perl does not enforce private and public parts of its modules as you may have been used to in other languages like C++, Ada, or Modula-17. Perl doesn't have an infatuation with enforced privacy. It would prefer that you stayed out of its living room because you weren't invited, not because it has a shotgun.

      The module and its user have a contract, part of which is common law, and part of which is "written". Part of the common law contract is that a module doesn't pollute any namespace it wasn't asked to. The written contract for the module (A.K.A. documentation) may make other provisions. But then you know when you use RedefineTheWorld that you're redefining the world and willing to take the consequences.

      IMHO, modules should not try to prevent calling or inherting their private methods. They should clearly document that it is a stupid or very stupid idea to do so, and that future implementation will break code that relies on behaviour or even existence of private methods.

      Also, modules using OOP should not have any private functions, but they should use methods documented as private instead. This way, I (as a user of the module) have the freedom to mess with the modules internals without resorting to monkey-patching, simply by inheriting from the module and fix what looks broken.

      Yes, this is against the basic ideas of OOP. No other class or object should mess with the "private" attributes or methods of an object, and only an exclusive circle of classes should be allowed to mess with "protected" attributes or methods.

      BUT: In the real world (as opposed to people drawing inheritance diagrams all day and all night), you sometimes have to break the rules to get things done. (See Re: DBD::CSV - how to I coax it to read BOM prefixed files? for a real-world example.)

      Alexander

      --
      Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

        IMHO, modules should not try to prevent calling or inherting their private methods. They should clearly document that it is a stupid or very stupid idea to do so [...]

        [...] This way, I (as a user of the module) have the freedom to mess with the modules internals without resorting to monkey-patching, simply by inheriting from the module and fix what looks broken.

        So I, as a module author, should provide an interface which people might need to rely on to accomplish what they need, and document that it's stupid to use it? That might be less work for me in terms of ensuring API stability, but for end users, I think they'd prefer I documented those methods (however obscure they might be) and committed to at least attempting to keep the API stable.

        Yes, this is against the basic ideas of OOP. No other class or object should mess with the "private" attributes or methods of an object, and only an exclusive circle of classes should be allowed to mess with "protected" attributes or methods.

        The reasons for this are not just theoretical purity; there are very good reasons for classes to have no undocumented methods. And subclassing is the biggest one of those. If I'm writing a subclass of your class, I might want to override some of your methods deliberately. But I sure don't want to override any of them accidentally, so I need to know what they're all called. That alone should warrant making anything you truly need to be private into a coderef.

      >  If it's a function/method which you have found useful/necessary, then other people might find it useful/necessary too, so make it part of your public API
      • I don't want to expose implementation details which will change in the future
      • I don't want to clutter the public API with obscure functions
      • I don't want to have compatibility discussions because I have to rename or drop an internal helper
      • I don't want the overhead to document something which is just a temporary hack for an early version / proof of concept.
      If someone finds a non API function useful after reading the source, he's still free to use it at own risk.

      The risk is obvious by the fact that it's NOT part of the public API.

      Occasionally I notice that internal functions might be useful for others, in this case outsourcing them to another module appears to be a better strategy.

      Cheers Rolf
      (addicted to the Perl Programming Language :)
      Wikisyntax for the Monastery FootballPerl is like chess, only without the dice

Re: (How) Do you document/test your private subroutines?
by afoken (Abbot) on Nov 07, 2018 at 00:03 UTC

    At work, we mostly use C and some C++. We use Doxygen to document it. All of the public stuff of a module is located in its header file, and so is the doxygen markdown for the public stuff. The private stuff and the implementation of public functions is in the C file, and we document it very much like the header file. That way, we can (hopefully) understand our code even in five years from now. Doxygen has the flags EXTRACT_ALL, EXTRACT_PRIVATE, and EXTRACT_STATIC that we set to YES to get the complete documentation. In some cases, the documentation of the public stuff is sufficient, so we can set those flags back to NO and get a much smaller documentation.

    For helper code (perl, bash, Windows batch files), we do roughly the same, often by using doxygen with a custom input filter.

    Code documentation is usally only a small part of our work. Medical and aerospace products require tons of paperwork, and the overhead grows with every new issue of the relevant standards. So documenting all of our code does not only help us maintain the code base, but also fulfils some obligatory requirements from the standards. And those of our clients that have bought the source code also get the complete documentation so they can modify the source on their own. Few clients actually do that, but they like the idea that they could.


    In my perl code, I usually document everything, simply because I rarely need to touch my code. Documentation clearly helps to understand what I did years ago.

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

      Thanks afoken, at work (mostly C++ fronted by Python), we use a couple of techniques to do similar what you've stated here.

      I'm also in an industry that has extreme regulatory (ahem: paperwork) oversight, and even though clients have no idea what they're doing, they claim they want x, y and z regarding documentation, so we try to provide it.

      Some of this falls through the cracks though, due to everyone wanting to fit the next upgrade into the current budget cycle (I digress on all of that, this is more personal/Open Source).

      I've never used Doxygen before. Looks very interesting, and I will read up on it. At a cursory glance, it looks handy already for a few of my open projects.

      I like the diverse responses here so far.

        I've never used Doxygen before. Looks very interesting, and I will read up on it. At a cursory glance, it looks handy already for a few of my open projects.

        A few more notes on doxygen:

        Doxygen comes with a very useful helper program called Doxywizard that allows to create a sane and well-documented Doxyfile with a few mouse clicks. Switching it to expert mode allows all kinds of fine-tuning. Plus, you can try all changes without actually changing the Doxyfile (Doxywizard feeds the configuration into doxygen from memory via STDIN).

        It is very recommended to install Graphviz. It creates useful (and very impressive) call graphs, class diagrams, and much more. All from normal code with doxygen comments, without any extra work (except for the computer). Our clients are regularily impressed by the amount of source code documentation that we generate. Most of that is automatically generated by doxygen and graphviz.

        We keep the Doxyfile in subversion and add a batch file or shell script named build-docs to automatically generate the documentation in HTML format in a directory docs just below the top-level directory of the project. Next to the script, we have a README.html that explains that doxygen needs to be installed, build-docs has to be started, and then links to where doxygen creates its output.

        The doxygen HTML output works much better with Javascript enabled. If you open the documentation from a filesystem (i.e. via a file:// URL), a modern browser will usually disable Javascript for security reasons. You may wish to add an exception or run a tiny webserver for viewing the documentation.

        Alexander

        --
        Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
Re: (How) Do you document/test your private subroutines?
by LanX (Archbishop) on Nov 06, 2018 at 23:32 UTC
    Concerning documentation:

    I once ran into the need to document my private methods and just included "inofficial" POD.

    The trick is to generate a Module.pod file for the "official" pod by filtering the Module.pm.

    Running "perldoc Module" will only show the .pod if present, and that's what CPAN shows.

    But running "perldoc PATH/Module.pm" will show the privat parts.

    I just included this filter into my tests, like this the .pod was always up to date after running prove.

    Parsing POD is straightforward, how to select the official part is up to you.

    HTH! :)

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery FootballPerl is like chess, only without the dice

      This is a very interesting approach LanX, could you elaborate on the specifics on how you do/did this (ie. filter the pieces public/private)?

      Thanks,

      -stevieb

        Um ... it's been some time and I never "perfected" the hack. :)

        Have a look here https://github.com/LanX/Macro/tree/master/lib

        Macro.pod.tpl has a template (read hack)

        podbuilder.pl reads the template and Macro.pm and creates Macro.pod and README.pod

        update

        As you see it's a primitive pod parser which grabs some parts between =head directives till =cut.

        you could just filter out any underscored _private headline or use a =head1 Internal to mark them.

        (not sure why it's missing in the git-hub version)

        Cheers Rolf
        (addicted to the Perl Programming Language :)
        Wikisyntax for the Monastery FootballPerl is like chess, only without the dice

        One possibility:

        =pod =for comment Foo! Or: =begin comment Bar! =end comment =cut

        I also seem to have a dim memory of someone suggesting =begin internal for marking things that are actually still documentation, but internal docs, but I'm not sure when/where that was.

Re: (How) Do you document/test your private subroutines?
by dsheroh (Prior) on Nov 07, 2018 at 09:47 UTC
    Well, first off, I don't tend to use private methods all that much. I'll slap an underscore onto the start of a sub's name if I don't think it's likely to be useful outside of the current module, but I tend to think that hiding the internals is overrated, so I rarely do it, even at the "asking nicely" level. If I were working in a non-Perl language with a shotgun to enforce privacy, I doubt I'd ever use it.

    When I do use private(ish) methods, I generally don't include them in any documentation, but "private" vs. "public" is not a factor in my testing. I test equally either way because my primary use for testing is to help me debug (present or future) problems and I find it useful to be able to localize the problem I'm debugging as quickly as possible. If A calls _B calls _C calls _D and _C breaks, I want my tests to be able to immediately tell me "there's a problem in _C", not "there's a problem in A" and then I have to manually look at A to see _B failed, then look at _B to see that _C failed, and probably also give _C a thorough going-over to be sure that _D didn't fail.

Re: (How) Do you document/test your private subroutines?
by harangzsolt33 (Scribe) on Nov 07, 2018 at 00:15 UTC
    I describe each sub, and then put a few comments here and there. But when I need to use this sub somewhere in a program, I usually just include a condensed version.
    ################################################# Trim # # This function removes whitespace, newline # characters and other special characters # before and after STRING. Returns a new string. # # Usage: STRING = Trim(STRING) # sub Trim { my $T = $_[0]; defined $T or return ''; my $N = length($T); my $X = 0; # ptr to first non-whitespace my $Y = 0; # ptr to last non-whitespace while ($N--) { if (vec($T, $N, 8) > 32) { $X = $N; $Y or $Y = $N + 1; } } return substr($T, $X, $Y - $X); }

    Condensed version:

    sub Trim { my $T = $_[0]; defined $T or return ''; my $N = length($T); my $X = 0; my $Y = 0; while ($N--) { if (vec($T, $N, 8) > 32) { $X = $N; $Y or $Y = $N + 1; } } return substr($T, $X, $Y - $X); }

      Thanks harangzsolt33, hang in there young Monk.

      Listen, observe, criticize, question and think about everything you see and read by those more experienced, all the while posting your thoughts and ideas. That's how we all begin and learn.

      Your process here, although valiant and commendable, introduces some obscurity relative to what I was asking, and the design of your post is more for a single developer, not a team or crew of developers/collaborators.

      Stay with it. Ask questions. Answer questions. Bang your head on your desk periodically. All real devs do these things :D

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1225330]
Approved by beech
Front-paged by haukex
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (6)
As of 2018-11-21 00:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My code is most likely broken because:
















    Results (234 votes). Check out past polls.

    Notices?