When is it better to NOT release a new module?

[snowhare]

I've been working on a new module (tentatively named Sub::Parms).

use Sub::Parms qw (:ignorecase :validate);

  sub some_method {
      MethodParms := $self, %args;
      my ($field_value, $another_field_value) = @args{'field_value','a
+nother_field'};

  }
 sub some_function {
      ParmsHash := %args;
      my ($field_value, $another_field_value) = @args{'field_value','a
+nother_field'};

  }


sub _legal_thing {
    unless (defined ($_[0]))      { return (0. 'Not defined');    }
    unless (int($_[0]) eq $_[0])) { return (0, 'Not an integer'); }
    return 1;
}

sub bind_parms_example {
 BindParms : (
   my $handle         := handle  [required, is_defined, can=param];
   my $thing          := thing   [optional, isa=CGI::Minimal];
   my $another_thing  := another [optional, type=SCALAR, validate=_leg
+al_thing];
   my $yathing        := yathing [optional, is_defined];
   my $defaulted      := dthing  [optional, default="help me"];
  )

    # .....
 
}

The problem is, it solves the same problem as a half-dozen other modules do: How do you simply, quickly and elegantly parse named parameters passed to a function/method?

I think my module brings something new and useful to the problem (why else would anyone ever write a module?): It provides clarity in code, is much faster (3 to 4 times) than any other module that solves the same problem and provides a simple way to turn off parameter validation assertions for a module in mature code to get even more performance out of it.

But is that enough to justify yet another module solving the same problem?

Or should I just file it in Acme::* for people's entertainment and edification?

[BrowserUk]

Here's a few reasons:

1. When you are doing so in order to "have a presence on CPAN".
2. When you have not utilised the module for at least two, different, real projects that make substantial use of it.
3. When you are not in a position to test it on at least 2 disparate platforms.
4. When you are unlikely to ever take the module beyond the first flush of interest.

   Ie. It will forever remain as version 0.xxx so that you can get away with a disclaimer about "use at you own risk".

5. When you are entirely unconvinced by the process and procedures that make up what might loosely be termed the CPAN requirements.

With a few more minutes thought I could probably come up with another half dozen.

Of course, most of these fly in the face of Perl community/OSS doctrines that suggest, amongst other things, publishing early and often. And this post will probably be seen as a rail against those doctrines--it isn't--and down-voted accordingly--which is fine.

Of the modules that I have written for my own purposes that might be (IMO) suitable candidates for CPAN, I have either:

• Lost interest and therefore would resent spending time maintaining them.
• Found problems during my own testing and use that I have yet to find a good solution to.

  In one case, the premise of the module was that it would do what it does in a substantially more space efficient manner than the 'normal' mechanism, whilst retaining a reasonable level of performance (kind of the use less 'memory'; idea). Whilst it is more space efficient, and early in testing, it showed good promise for performance, once I started tackling the edge cases, adding the extra overheads of validation, and handling a full range of binary data, it meant that the performance fell off markedly.

  For my own purposes, I am, prepared to live without the edge case detection to benefit from the performance, but having written the code, I understand its limitations in use. For the general case, even documenting those limitations is not sufficient (IMO) to make the code usable in production environments, and with the edge case detection, the performance drops to the point where the benefits are questionable.

  In another case, the module functions fine, nine times out of 10, but on the 10th occasion it will throw a wobbly. Again, for my purpose, ie. the purpose for which I originally wrote it, it functioned fine and allowed me to do what I needed to do, but for it to be usable as a general purpose tool (in an already well represented category) I would need to find and correct that flaw.

  Given there are several existing tools that are well known, well used and well supported (including a core module), the major benefit mine has over those others is a niche requirement. So, I had to ask myself whether I was prepared for the work involved to bring the module up to the standards that I would like to expect from a module on CPAN. And would that effort be of great benefit in an already crowded market. My conclusion was no.

• Finally, there is an expectation of compliance with certain ways of presenting and delivering CPAN modules. I'm not referring to quality specifically, but more "the way things are done".

  I have a strong distaste for several of these ways of working. My objections are neither frivolous nor contrary. They are based upon my considered opinion formed over many years as a coder, and upon my few years of using Perl. Individually, no one of them would prevent me from complying with the 'standard practice', but cumulatively, they amount to an intense and constant irritation when I have to perpetuate them.

  I have expressed several of my reservations here in various posts over the last 3 1/2 years, and gauged the usually negative, often hostile, sometimes knee-jerk, reaction to them. I have, accordingly, re-considered my position on them several times. And mostly, each time reached the same position as I started with.

  Hence, I've decided that it simply isn't worth the perpetual irritation of complying with working practices with which I do not agree, for stuff I do as a hobby. Having spent the best part of 25 years complying with other peoples (often ill-judged) coding standards and working practices in order to earn a living, I feel no incentive to do so with stuff I write for fun, for my own education, and in the hope that it may occasionally be useful to others.

  In a democracy, the majority is always right. In my free-time, I am a majority of one.

  In the workplace, the boss, or the client, or the customer or entrenched dogma is right. The guy paying the money is the final arbiter and you choose your position and suffer your irritations accordingly. Working alone, in my own home, and in my own time, I do not have to suffer those irritations, or bend my position to the will of those others.

  This changes when in paid employment, or even unpaid collaboration. Though in the latter case, I would anticipate at least a hearing on an equal footing, before submerging my own opinions and preferences in favour of those of another.

So, when you make your decision about whether to publish your module, which to me seems a perfectly reasonable approach to a perpetual thorn in Perl's side--it's lack of compile-time signature checks combined with either repetitive manual parameter checking, or high overhead automated checks--consider your position carefully.

Given Perl's already tardy sub/method call performance, anything, that moves some of the overheads to compile-time has to be worthy of serious consideration. Even if it does rely upon a coding technique that is universally scorned because it imposes a need for a high level of care and testing upon the author and may impose an extra degree of care and limitation upon it users. That extra care will mostly relate to their choice of the forms of syntax and source code formatting they use. As your module is hardly likely to see great use in obfus or golf games, which are the only places that the most quirky and difficult areas of Perl's syntax usually see light of day, most production code should already be formatted to a level of cleanliness that should avoid most of the traps that befoul source-filters.

Assuming you exercise appropriate care in its construction and testing :)

Consider your position in the knowledge that, whilst PM is generally a tolerant and happy place to coexist, the wider Perl community has it's dogmas and can be pretty ruthlessly intolerant of even well considered contrary opinion to them. And, within that wider community, whilst there are several camps, of loosely grouped subsets of opinion, there are also some areas where the entire group will come together to ... what was that word I saw someone use recently... kibitz those that voice opinions contrary to the established dogmas.

Good luck with your module, and if possible, I would like to have a play with a copy--even if you decide not to make it generally available.

[rinceWind]

I think my module brings something new and useful to the problem (why else would anyone ever write a module?): It provides clarity in code, is much faster (3 to 4 times) than any other module that solves the same problem and provides a simple way to turn off parameter validation assertions for a module in mature code to get even more performance out of it.

These are your selling points. This is what you should stress in your pod, if you are going to release it as a module.

You should also present some evidence, by running benchmarks against Params::Validate and whichever other modules you are wanting to compare with.

If it's that much better, go ahead and release it. If it's comparable to an existing module, work out how you would convince others to use your code instead.

[Perl Mouse]

The problem is, it solves the same problem as a half-dozen other modules do: How do you simply, quickly and elegantly parse named parameters passed to a function/method?

The fact that half a dozen other modules do the same thing means that those modules solve the problem poorly in some peoples opinion. In fact, enough people found the solutions poor that the module got re-invented over and over again.

But is that enough to justify yet another module solving the same problem?

I don't understand the question. You already created the other module, and you like it - so it *did* justify another module. Otherwise, you wouldn't have done it (or not have liked it). Put it on CPAN. It only takes a single other person to like it for justification.

[bluto]

The fact that half a dozen other modules do the same thing means that those modules solve the problem poorly in some peoples opinion. In fact, enough people found the solutions poor that the module got re-invented over and over again.

Perhaps it's just another manifestation of caring about the color of the bikeshed. The sheer number of config file parsers on CPAN proves this is a primal instinct. :-)

[perrin]

If it were a templating module, I would say no. This area is somewhat less flooded, so the barrier to entry shouldn't be as high. However, I would urge you to make it clear why someone would choose this module over the established standard, Params::Validate. I don't see much difference here beyond syntax, and frankly, your syntax is pretty bizarre by comparison to that module's simple data structure spec. Is it really faster than Params::ValidateXS?

[Tanktalus]

Make it clear why someone would choose this module over other modules, yes. But, if you can, also try to make it clear why one would not choose this module - what limitations or other downsides does it have? For example, some people have a perverse fear of source filters (others have a very reasonable fear of source filters ;-}), so because you use this controversial feature, you should clarify that. For example, I'm not sure, but somehow I think you'll have problems inside an eval STR. Maybe not - but the better you document what you can and cannot do, the more realistic users' expectations will be.

Even if it means that I need to avoid doing something so that your module works, that's still relevant information. (And if you plan on cleaning up that limitation in the future.)

[snowhare]

Substantially faster.

Read more... benchmark-arg-parsers.pl (4 kB)

                                       Rate
simple_parms                        28329/s
params_validate                     48077/s
cf_standard_args                    73529/s
sub_parms_bind_parms_function_req  157480/s
sub_parms_bind_parms_function      172414/s
one_step_args                      224719/s
standard_args                      229885/s
sub_parms_method                   238095/s
sub_parms_function                 270270/s
positional_args                    952381/s
null_method                       1666667/s
null_sub                          1666667/s

[wazoox]

Well, Free Software is all about choice... Publish your module, don't be shy! If there are several providing -roughly- the same service, competition should help the fittest to survive :)

[szbalint]

Actually I was pondering about the same thing...

I've written a BBCode parser, which is quite fast:

               Rate     BBCode::Parser  HTML::BBCode     Method One   
+  Method Two
BBCode::Parser 2.54/s             --          -92%           -98%     
+      -99%
HTML::BBCode   33.0/s          1200%            --           -80%     
+      -82%
Method One      161/s          6252%          389%             --     
+      -11%
Method Two      182/s          7060%          451%            13%     
+        --



Method One and Method Two is the same module, but the second is optimized.

Would it be worth to publish it? I'm afraid the code is not well documented at all though.

Is code runtime speed important in the case of a BBCode parser, anyway?

[Aristotle]

What about working your routine into one of the published modules (or both) and sending the author(s) a patch? He might also decide he's lost interest in maintaining the module in the meantime and hand you the reigns.

[xdg]

The problem is, it solves the same problem as a half-dozen other modules do... I think my module brings something new and useful to the problem

I strongly suggest reading the excellent comparison to other modules found in Tree::Simple. If you can write a similar piece and you feel that your module stands up to scrutiny once you've done that, then I say go ahead and publish it.

[diotalevi]

Your module appears to be a source code filter operating on general purpose perl. Such things should never be released. They're evil, nasty things and cause problems for other people.

[BUU]

Your module appears to be a source code filter operating on general purpose perl. Such things should never be released. They're evil, nasty things and cause problems for other people.


Ya know, I too once had this opinion. I even asked Ovid about it. I felt his response was enlightening enough to reproduce here:


Now that I have the lying out of the way, I can understand the hesitation about them. I worked for Rentrak for a year and they did something very similar (but on a much smaller scope) to what I'm trying to do, but they did it with Filter::Util::Call. Naturally, when I first started working with this I complained heartily to my programming friends about using a source filter in production code. And not just any old production code, either. We're talking about code that is rightfully called "enterprise class." And you know what? In the entire time I used it, I only stumbled across one bug (and admittedly it was a bugger to track down.) I know what the bug is and it's not going to be in my version :)

So after working with a source filter for over a year, what did I discover? Source filters, when done properly, can make programming much, much easier. I didn't realize how much I missed signatures until I had them again. I also didn't realize how much I missed having an entire class of bugs pretty much disappear because I had signatures. For the one bug that was caused by this source filter, programming was much more productive and bug free.

I was wrong about source filters. I used to think they were terribly evil and should not be used, but now I realize (as with all dogmatism) that this is not universally the case. Admittedly, source filters generally need to be some of the most heavily tested code, but the payoff can be huge.

But serious programmers don't use source filters, right? Don't tell that to Damian and Ingy, two well-known and respected programmers who are quite happy to reach for source filters when that's the solution to a thorny problem. Those, naturally, are two that come to mind but there are plenty of others. Source filters are to be used sparingly, but it's not the case that they should never be used. They are not evil, just dangerous.



This was taken from his journal at: http://use.perl.org/~Ovid/journal/22152. The following discussion also links to this node: Re: Use method/function signatures with Perl which discusses it further.

[diotalevi]

[Edited Its wrong of me to characterize everything done by a particular author this way. There are a small set of examples of bugs in his published work and some things like years passing while bug fixes submitted to rt.cpan.org aren't applied.] Ingy is no example. Bugs in his work are unsurprising and regular. I don't know whether this is related to his use of source filters or not.

I checked Damian's list of categorized modules for stuff that used something from Filter. Here's how it breaks down for his source filtering modules. It works out that while Damian has written several things using source filters, he has only one thing that isn't marked experimental that actually uses source filters that would be available in production. [Edited He writes some fabulous stuff but there isn't any evidence that he's written anything using source filters that he considers worthy of running in production.]

Modules you aren't supposed to use anyway

• Lingua-Romana-Perligata
• Perl6-Currying
• Perl6-Export
• Perl6-Placeholders
• Perl6-Rules
• Switch

Attribute-Handlers-Prospective is the only module that was superceded by other things. You shouldn't use it.

He said Inline-Files is production worthy but its marked as experimental in its pod?

He uses Smart-Comments and Toolkit as development aides. It turns out that you aren't supposed to use Smart-Comments in production anyway - its a design feature that they're easy to remove. So production code doesn't even touch a source filter.

Regarding Ovid's code. Its much easier to have a source filter when you're running it against code you control and with some known code standards. Its much less sane to do that when you're releasing things onto CPAN where it will have to deal with anything that's potentially available.

---

I assert that Ovid's anecdotal evidence does not indicate that source filters are ok for general use on CPAN.

[snowhare]

You are right that it is a source code filter (albeit one that is portable all the way back to at least perl 5.005_04). But would you extend your remarks on why you believe that that of itself is a good reason to not release it? I have tried to deliberately create a syntax that cannot be easily 'triggered' by accident or mis-parse in a way that generates false line numbered errors. If a particular line in the original causes an error - that will be the line the compiler reports after processing as having generated the error. And a module has to intentionally use it - so it isn't like a random module will be "side-swiped" by it.

So what, specifically, do you find objectionable about the use of source code filters?

[pdcawley]

Your module appears to be a source code filter operating on general purpose perl. Such things should never be released. They're evil, nasty things and cause problems for other people.

Tosh! If that were really the case, why is Perl 6 getting some seriously powerful tools for writing source filters? Source filters are hard to do well because general Perl is hard to parse, but that's a very long way from saying that they are evil and should never be released. Looking at the syntax shown in the module in question (and the performance figures claimed), I'd certainly be interested in using it in my own work.

[Aristotle]

If that were really the case, why is Perl 6 getting some seriously powerful tools for writing source filters?

Mu.

They are evil in Perl5-land because they are, for all intents and purposes, impossible to make work reliably. They will be acceptable and accepted in Perl6-land because they’ll be easy to make work reliably.

Personally, I’m far more excited about smoothly integrated macros than reliable source filters.

[diotalevi]

I rather expect the perl6 toolkit is more macro-like in intention and that's why its so cool. Of course I want the power of a source filter - its just that there's an acceptable implementation and unacceptable things like the OP suggested.

[jacques]

When is it better to NOT release a new module?

When you can't handle criticism of your work.

[tirwhan]

When you can't handle criticism of your work.

Or the incessant flood of thank-you notes, patches, donations and offers of employment that invariably follow the release of a CPAN module.

[Aristotle]

The occasional offer of reproductive duties makes the flood worth sifting through, though. ;-)

[tilly]

I am trying to decide whether you're being naive or sarcastic.

[dragonchild]

An alternative is to provide patches to existing modules. Or, provide a compat layer for others who may wish to transition. I did the former with PDF::Template and it was very well received.