Re^4: An Introduction to Literate Programming with perlWEB

[doom]

Well, this is what I would tell you about words and code:

Unless you're working from a fixed spec (which is more the exception than the rule, I'd say) the API of a module is going to change as you're working on it. If you keep the pod describing what a sub does in the same file as the sub, I think you're much more likely to remember to revise the docs when the API changes.

Further, it's often a very good idea to use some comments throughout the code. The "paragraph style" works well: a "topic sentence" in english, followed by detailed explication in the form of code. Comments at the end of a line of code are good places for things like TODO notes and even hints to perl beginners ("hash slice", "schwartzian transform").

As for things like this:

The code is description of the actual algorithms used and their implementation. And should be the only such description. It cares not for the external abstraction.

My personal opinion is that techies really need to watch these kind of religious beliefs -- we're always trying to squeeze the world into these neat, idealized doctrines, but the world always fights back. For example, if you were to take what BrowserUK is saying seriously, you would insist that perlguts should not exist.

[BrowserUk]

For example, if you were to take what BrowserUK is saying seriously, you would insist that perlguts should not exist.

Au contraire!

perlguts is a great example, but for exactly the opposite reasons to those you seem to think. It doesn't describe an implementation; the names of varibles used for loop counters; or the algorithm used (for example) to calculate the hash of a key for an associative array.

It is user documentation describing an API.

In this case, the users aren't Perl programmers, they are XS programmers. It is, in part, derived from embeded, interleaved tags (in the manner of JavaDoc or Doxegen). But not from embedded prose re-describing the internal implementation. XS is an abstract API, implemented through C macros, and perlguts documents that API at that level of abstraction.

And to reinforce the power of that abstraction, when Dave Mitchell re-implemented large chunks of the underlying code for 5.10.0, (actually 5.9.something). to reduce the memory footprint of many of the internal structures; Perlguts hardly changed at all. Your own example makes my points above more strongly than I ever have.

There's nothing "religious", no "neat, idealized doctrines" involved. Just simple, practical, proven methodology derived from hard won experience. Not my ideas, nor my experience, but that of 50 years of those that went before us.

[doom]

Okay, you have a point, but my claim still stands: there is nothing wrong with documenting internals, and that's certainly one of the ways in which perlguts is used: it's recommended reading for beginning perl-porters.

Will the actual state of the code drift from the internals documentation? Yes, certainly, but that doesn't make the docs useless-- at the very least they tell you something about where the codebase was at an earlier stage. When you're getting used to an unfamiliar code base, any hints at all are worthwhile.

And I would suggest that what the perlguts example really illustrates is that the distinction between internal and external is fuzzy, because that boundary moves around depending on what you're doing -- to an XS programmer, perlguts is the API, to me, it documents some internals I don't need to think about just now.

Note that if you put every routine into it's own module with it's own API to document, then there would be little difference between your position and mine.

There's nothing "religious", no "neat, idealized doctrines" involved. Just simple, practical, proven methodology derived from hard won experience.

That's what they all say.

Not my ideas, nor my experience, but that of 50 years of those that went before us.

You mean the experience of guys like Donald Knuth?

[BrowserUk]

When you're getting used to an unfamiliar code base, any hints at all are worthwhile.

If you put every routine into it's own module with it's own API to document, then there would be little difference between your position and mine.

On any project that runs to more than half a dozen files, there is absolutely a need for project documentation, separate from, and with a different target audience to User/API documentation. Targeted at maintainers, this should include such things as file layouts; internal structure layouts; the structure of the project directories and files; build, test and validation procedures etc.

But when you're first sitting down to explore and familiarise yourself with a project of this size, the last thing you want to have to do is stab around in the dark peeking inside source files at random trying to piece together an overview by picking out those bits of the embedded prose that relate to that overview, from those bits of that embedded prose that relate only to the local line, block, subroutine or file.

Not only does that make it difficult to form a coherent overview of the project initially. That difficulty persists throughout the life of the project for all concerned. Everyone--from programmers, to architects to managers to technical writers--waste an inordinate amount of time trying to remember not just what they read, but where they read it. It is all but impossible to index the information at any given level it for any target subset of users.

This type of project overview or architectural site map, absolutely does not belong spread across myriad source files mixed up with "this loop counter is unusual because it starts at -1" type local annotations.

You mean the experience of guys like Donald Knuth?

Hm. He has seriously long and deep experience yes, but of a very different kind to your average Joe Programmer, systems analyst or project architect.

It all comes back to my short answer in my original reply above.

His uses and needs of a documentation system are far and away different to those of your working stiff programmer or project leaders. He's targeting an entirely different type of audience; his goals are: absolute technical correctness, and understandability by first-year undergrads regardless of background; his timescales are measured in years, if not decades, rather than the days or weeks as for most commercial projects.

This puts me in mind of a heated discussion I had a dozen years ago with the head of HR for an extremely large organisation. Try as hard as I might, I could not convince her that two years of relevant experience was far superior to ten or more years of the irrelevant.