Perldocs and peer reviews

[g0n]

With regard to this thread about CGI::Prototype, and this thread about 'non-serious' modules, the issue of code QA seems to be coming up quite a bit lately.

While there is a framework in place for testing and peer review of modules, fully testing a module can be a very time consuming undertaking. Once you have

• Downloaded the module
• Resolved the dependencies
• Installed the module
• Familiarised yourself with the purpose and interfaces
• Written code to test out the module
• Reported on any problems

You've put quite a lot of time and effort in.

As an alternative for those who don't have the available time, can I suggest a second tier of testing? It would take minutes only to briefly review the perldoc only of a module, and report back to the author where

• the docs are unclear
• the docs are incomplete or sketchy
• the described syntax is nonintuitive or unfriendly
• (any other appropriate guidance or suggestions...)

To skim through the docs for an average module on CPAN, without attempting to download, install and run the code, could take perhaps 15 minutes to half an hour. I think many more people could spare that amount of time to review new or amended modules.

At the very least, as well as improving the quality of documentation, it would be a second pair of eyes looking at the design assumptions, which can only be a help.

If anyone agrees, perhaps this would be a good place to discuss what specific elements to look for in a review?

Charles

Update: Several people have raised the concern that this potentially would result in low quality feedback from people who have not used a module and hence may not understand what it is seeking to do. I should stress that this suggestion is intended to widen the range of modules that get some kind of review, rather than add to the volume of feedback on mature & widely used modules.

The following notes would be my suggested starting point:

• Only perform a POD review on modules that do not have a cpan rating
• State at the beginning of any feedback that this is a POD review only
• Only review modules that relate to an area in which you have some knowledge
• Don't necessarily expect an answer, and do not follow up your communication unless invited to do so
• Try to give suggestions rather than just criticism

[xdg]

I agree that perldoc feedback is a good thing, but I'm not sure that a formal framework is necessary. If I find an error or some slight ambiguity in the documentation of a module, I email the author or put a bug report on the bug tracker. But if a module's documentation just stinks in the first place and shows little concern for the needs of users, it doesn't make me very hopeful that a documentation review is going to make any difference unless I'm actually sending a full patch for the pod.

(Maybe some authors hope that other people will like the module so much that they'll document it for him/her. That might explain the proliferation of real documentation showing up on wikis for several major frameworks. But I don't think that's very good practice if its intentional.)

[dragonchild]

It's not intentional. When I wrote Excel::Template, I was writing it for an employer. I put it on CPAN to demonstrate ownership, but didn't have the time to document it. Later, and thankfully so, others have provided documentation questions and/or patches. Otherwise, I would always be waiting for those magical tuits to appear ...

[theorbtwo]

The problem with that is that it's possible for the docs to be clear, seem complete, and describe friendly syntax... and be wrong. The docs for Math::Matrix, for example, fail to describe a very useful feature -- that the objects have a number of overloaded operators. Somebody just reading the docs and not the code wouldn't be able to tell. (Reminds me; I never got around to writing a bug-report on that.)

It's possible for modules to even have example code in their docs that doesn't run -- something undetectable without actually installing the module. This is what Pod::Tests and other similar modules are for, but not everybody (myself included) uses them.

[g0n]

The problem with that is that it's possible for the docs to be clear, seem complete, and describe friendly syntax... and be wrong

Which of course a doc review would not highlight. But normal testing and use would.
I'm not advocating POD review as a replacement for functional testing, but as a way to get more people involved in distribution QA in a way that is easier and more accessible than performing full blown testing, and getting a 'first pass' done on a wider range of modules.
After all, full testing on every module on CPAN is well nigh impossible, but if we could get a POD review on a wider range, that would be better than nothing.

[Anonymous Monk]

As a CPAN author, I appreciate feedback from users. But I'm not waiting for people who after "skimming" the docs, tell me they find them unclear, or find the the described syntax nonintuitive or unfriendly.

Really, I spend a lot of time working on my code and the documentation. The least I expect of a user is to spend a little time with the module before critizing it. What might not be clear after just skimming the docs, might make a hell of a lot of sense if you actually use the module.

Feedback is good, but bad feedback is worse than no feedback.

[g0n]

Mm, not an aspect I'd considered, I must admit.

However, I'm not sure I agree that bad feedback is much worse than no feedback. A trollish or daft feedback mail can just be binned. How much notice to take of feedback is a judgement call on the part of the author. If you are getting helpful feedback from a range of regular users, then a POD skim is not likely to be helpful.

On the other hand, for those of us (raises a timid hand) maintaining 'niche' or new modules, which don't have a significant user base, feedback saying 'xyz is not really clear/your constructor syntax is difficult to follow' etc might be of help.

I come back to the comment earlier in the thread - I'm not suggesting that we replace the process of CPAN testing, but widen the range of QA by looking at the POD of distributions that otherwise might not go through any other public testing.

[Aristotle]

A trollish or daft feedback mail can just be binned.

That doesn't mean we should encourage people to send more superficial feedback in the hopes that some of it will be useful, though.

[Anonymous Monk]

A trollish or daft feedback mail can just be binned.

Yeah, and so can spam. Trollish or daft feedback still require the author to read the reply. It's still a nuisance and it disrupts the process.

feedback saying 'xyz is not really clear/your constructor syntax is difficult to follow' etc might be of help.

might be is not good enough. You might be helped by such feedback, but I certainly am not. Unless you can specify why it's not clear or difficult to follow, and what the alternatives could be, I don't know whether you simply don't know Perl very well ("not clear" and "difficult" is subjective) or perhaps you do, but you don't really know the module I wrote very well and what you think would be simpler would mean loss of functionality.

That's bad feedback, and is, IMO, worse than no feedback. It's just like questions. A wrong answer is worse than no answer at all.

[Ardemus]

My comment here is a good example of people's concerns. I read the comments and I have an opinion. However, I have never written a POD or a module.

A full knowledge of a subject often changes my opinion of it. Take the American Electoral Colledge. Once I read about the history and reasoning behind the system, I reversed my opinions on it's validity. I think that far too many people strongly believe their uneducated opinions.

That said, I support the *idea* of documentation review. I suggest that documentation should make the module's purpose clear. The idea that using the module will clarify the documentation seems backwards. On the other hand, the mechanical details may be best learned hands on, with the docs as a guide.

Perhaps a simple questionaire would be appropriate / helpful. Asking questions like:

1. What is the primary purpose of this module?:
   • Common answer A
   • Common answer B
   • Common answer C
   • Recent answer A
   • Recent answer B
   • Recent answer C
   • Enter your own

2. "What are the core features of this module?"
3. "Is the language of this module easy to follow?"
4. Etc.

This could be aggregated into a format that is easy for authors to review.

In any case, just brainstorming.

[g0n]

Opinion is divided on the question of a structure. XDG earlier in this thread commented that a structure was possibly unnecessary. I lean more towards your view. A structure, however loose, would provide guidance to reviewers and probably make potential reviewers more willing to comment without a) generating poor quality feedback, or b) putting people off reviewing because they don't want to send poor quality feedback.

Here's a few possible questions:

• Does the synopsis clearly explain what the module does?
• Is the syntax of the constructor and any main methods clear from the perldoc
• Is there any functionality not documented (and therefore probably not present) you would like to see in the module
• Is the perldoc correctly formatted (and if not, what is the problem and where)?
• Does the meta data contain a) license, b) language used c) development stage?

As far as brainstorming goes, it might be worth considering opt-in/opt-out of perldoc reviews, setting up a site to post to rather than direct to the author etc. The danger there is that we then potentially start talking about review activity taking place somewhere other than CPAN, which I'm a little dubious about.

c.

Update: Then again, I just looked at recently active threads and found this

[zby]

I do believe we need a more lightweight way for feedback on CPAN modules, not only on their documentation. Once there was a wiki for all modules created by RIP Iain Truskett. And wiki is perfect for this goal - it is both lightweight and document based (in stead of message based Perl Monks). Unfortunately that site died shortly after it's creator.

[g0n]

I guess its time to put my money where my mouth is. Since I advocated this scheme in the first place, its only fair that I should volunteer as guinea pig.

I've uploaded version 0.14 of AI-NNFlex to CPAN here, and set up a cpanforum here. I'd like to invite anyone and everyone to comment, including perldoc review feedback of the sort we've discussed in this thread.