Re^3: An Introduction to Literate Programming with perlWEB

[BrowserUk]

BrowserUK likes to complain about the need to maintain code and documentation in parallel, but that problem doesn't go away if you move the docs to another file

Oh, but it does!

When writing documentation in a different file, people are far less tempted to start describing how the code works--opens a file; increments a variable; sorts an array; stuff which all but the most novice of programmers can (and if they are to do a descent job of maintenance work; should), work out for themselves, from the code.

Instead, they (should) concentrate on:

• the what: the public APIs and their parameters;
• the why: their function and purpose, in terms of what they are intended to do on behalf of the calling code;
• the usage: this one is far to often glossed over or omitted completely.
• the contract of that API: its requirements of the caller, and promises to them.

Documentation is for users; and should be written at the same level of abstraction that the user will use the exported api.

The code is description of the actual algorithms used and their implementation. And should be the only such description. It cares not for the external abstraction.

You should be able to re-implement the internals of an published (documented) API, using different internal algorithms, or even a different language, and the documentation should not need to be change at all. Nothing in the documentation should need to change, when the implementation changes, provided that the published API is maintained.

The purpose of comments is to annotate the code with additional (brief) information, pertinent to that code, that the language does not allow to be conveyed easily by the code itself. It should not repeat the code; nor the documentation; nor the language manuals.

[doom]

Well, this is what I would tell you about words and code:

Unless you're working from a fixed spec (which is more the exception than the rule, I'd say) the API of a module is going to change as you're working on it. If you keep the pod describing what a sub does in the same file as the sub, I think you're much more likely to remember to revise the docs when the API changes.

Further, it's often a very good idea to use some comments throughout the code. The "paragraph style" works well: a "topic sentence" in english, followed by detailed explication in the form of code. Comments at the end of a line of code are good places for things like TODO notes and even hints to perl beginners ("hash slice", "schwartzian transform").

As for things like this:

The code is description of the actual algorithms used and their implementation. And should be the only such description. It cares not for the external abstraction.

My personal opinion is that techies really need to watch these kind of religious beliefs -- we're always trying to squeeze the world into these neat, idealized doctrines, but the world always fights back. For example, if you were to take what BrowserUK is saying seriously, you would insist that perlguts should not exist.

[BrowserUk]

For example, if you were to take what BrowserUK is saying seriously, you would insist that perlguts should not exist.

Au contraire!

perlguts is a great example, but for exactly the opposite reasons to those you seem to think. It doesn't describe an implementation; the names of varibles used for loop counters; or the algorithm used (for example) to calculate the hash of a key for an associative array.

It is user documentation describing an API.

In this case, the users aren't Perl programmers, they are XS programmers. It is, in part, derived from embeded, interleaved tags (in the manner of JavaDoc or Doxegen). But not from embedded prose re-describing the internal implementation. XS is an abstract API, implemented through C macros, and perlguts documents that API at that level of abstraction.

And to reinforce the power of that abstraction, when Dave Mitchell re-implemented large chunks of the underlying code for 5.10.0, (actually 5.9.something). to reduce the memory footprint of many of the internal structures; Perlguts hardly changed at all. Your own example makes my points above more strongly than I ever have.

There's nothing "religious", no "neat, idealized doctrines" involved. Just simple, practical, proven methodology derived from hard won experience. Not my ideas, nor my experience, but that of 50 years of those that went before us.

[doom]

Okay, you have a point, but my claim still stands: there is nothing wrong with documenting internals, and that's certainly one of the ways in which perlguts is used: it's recommended reading for beginning perl-porters.

Will the actual state of the code drift from the internals documentation? Yes, certainly, but that doesn't make the docs useless-- at the very least they tell you something about where the codebase was at an earlier stage. When you're getting used to an unfamiliar code base, any hints at all are worthwhile.

And I would suggest that what the perlguts example really illustrates is that the distinction between internal and external is fuzzy, because that boundary moves around depending on what you're doing -- to an XS programmer, perlguts is the API, to me, it documents some internals I don't need to think about just now.

Note that if you put every routine into it's own module with it's own API to document, then there would be little difference between your position and mine.

There's nothing "religious", no "neat, idealized doctrines" involved. Just simple, practical, proven methodology derived from hard won experience.

That's what they all say.

Not my ideas, nor my experience, but that of 50 years of those that went before us.

You mean the experience of guys like Donald Knuth?