Miscellaneous.pm?

[tlm]

Dear monks,

I have a module that is nothing but a miscellaneous collection of little subroutines that have nothing in common other than that I find them handy. (E.g. my collection includes a sub for stripping leading and trailing whitespace from a string, one for producing a standard timestamp string, a wrapper for CORE::open that returns the open handle and takes care of the error checking, a function that removes duplicates from a list, preserving order, etc., etc.)

I suppose many Perl programmers have some scheme for reusing such useful but isolated snippets. It is not a terribly thorny problem, but I am curious about the details. What's your scheme like? If it is a module (and what else could it be?), what do you call it? Unfortunately, the better the name, e.g. Utilities.pm, the greater the likelihood of a namespace collision. Do you include this module in your software distributions, or do you use it only during development? Have you considered submitting it to CPAN?

Moved from Seekers of Perl Wisdom to Meditations by planetscape

[TheDamian]

I wrote this to solve that very problem.

Damian

[xdg]

There was a recent, related thread here: Organizing personal perl library (AKA, personal CPAN)

[rhesa]

See also Cut and Paste programming, which warns:


Beware of Utility, Helper, Misc, etc packages. They collect orphan code. The pressure to move things out of them is very low, as they all seem to fit by virtue of not fitting anywhere else. They grow indefinitely in size because the class of things that don't seem to belong anywhere is very large. The effect snowballs as the growth of other objects is stymied while the "Utility" package booms.

[diotalevi]

My source is full of little sub uniq { my %seen; grep !$seen{$_}, @_ } functions, all alike. Things this small are too small to remember to include specifically so they get retyped whenever needed.

[markjugg]

Are you aware of List::MoreUtils? It contains a "uniq" function much like this.

BTW, there seems to be bug in your routine. I think you meant something like !$seen{$_}++.

That illustrates the value of having even a "simple" routine in a re-usable module!

[diotalevi]

That module is on my to-avoid list. It's poisoned with prototypes and is chock full of functions of dubious utility and quality. Thanks for the note about the bug in my response. I don't make the same mistake when I'm typing at source code.

In general, I'd prefer to inline these sorts of tasks anyway. It's only while experimenting that I use these as broken out functions.

[exussum0]

I really really suggest you not use a name like Misc or Utility. It is most likely mis-named.

I have a module that is nothing but a miscellaneous collection of little subroutines that have nothing in common other that I find them handy. (E.g. my collection includes a sub for stripping leading and trailing whitespace from a string, one for producing a standard timestamp string, a wrapper for CORE::open that returns the open handle and takes care of the error checking, a function that removes duplicates from a list, preserving order, etc., etc.)

Why not put them in libraries called String, Time and List respectively?

(rant to the general populous) The word utility, helper and misc denote very broad or unsually specific. The following is an absurd example to prove the point.

One wouldn't create a package called AllMethods that contains everything under the sun in CPAN. It's unweildy and undirected. What if your Utility package has 100 little utility functions. Then you need to refactor, which can be a huge pain depending on the number of developers and the tools available. Moving concepts around will also be hell. There would be too much shared data.

Neither would one create a package for each method. Requiring a user to use all appropriate packages would be a bear to manipulate data. Dependencies could be quite the nightmare. Removing an entire concept would be hell.

The resolution: do not try to avoid classifying a method into the appropriate package. A package w/ one method is not neccessarily evil as one w/ 100 methods as long as they show relation. Yes, a 1 method package may seem silly, but what happens with your time method when you decide to add another? You have an appropriate place for it now.

But what of misclassifications? One misnamed or misplaced funciton is not too hard to move around as long as due dilligence on not waiting until the works becomes too much.

Please. Don't make a Helper/Utility/Misc package. It leads to messy code or less convenient code. (/rant)

[CountZero]

Snippets in Komodo's toolbox.

[duckyd]

I am curious to see if this thread gets a solution that I like better than my own. I have one module around named along the lines of "Utilities.pm" that such funcitons end up in. Since my code has yet to end up on CPAN, I haven't had to worry about distribution issues. In a module bound for CPAN, I'd be wary of depending on such convenience functions - if they really are such little snippets, it's probably not worth adding another module to CPAN and depending on it. I'm not a big fan of such catch-all modules, so I'd rather see useful functions added to CPAN in some approrpiate (more-well-categorized) place.

If a namespace issue should crop up with such a module in my own code, well, perl -pi -e 's/.../.../' will be a pretty quick solution.

[tirwhan]

One thing to remember if you do put these functions into one or more modules, don't export your subroutines by default in @EXPORT, rather put them in @EXPORT_OK so that the importing code needs to import them explicitly. That way you avoid possible collisions with subroutines in other packages, but also (more importantly IMO) you make it obvious to the reader of your code where that function came from.

[Callum]

I keep a callum.pm module around which contains a variety of personal subs, and obviously this has never been intended to appear on CPAN or anywhere else public.

However it does feature in scripts I've written at work, and which may be seen or even adopted by others. In these cases the value of "callum.pm" over "misc.pm" becomes great -- at the very least providing a warning to adopters that it's a personal script.

Obviously things vary according to how public a script is, if it's something that only I will ever use then structure & conventions are essentially just for my benefit and names like sqlstuff.pm, googlestuff.pm even (ugggh!) miscstuff.pm abound; if "real" people are going to see or use the code then most modules will be standard cpan or corporate ones, those that aren't have sensible names, are structured, commented, and maybe even have some error handling ;)

In the middle lie the potential problems; like I suspect many of you, I have a variety of (for wont of a better term) utility scripts that a few others use or copy, and it's helpful in these cases to have a middle ground -- scripts I've thrown together, and that have grown organically (as in -- like mold) over a few years, and which are used by a variety of random techies, who hopefully understand the nature of the code, and who will in general tweak it to suit themselves better anyway.

Obviously this isn't best practice in any sense, but I see it as a reasonable middle ground between "proper" behaviour and the random-script-thrown-together reality.

[exussum0]

I agree, it's a perfectly reasonable middle ground etc etc solution on one condition. Only one. It never gets distributed and if it does, it gets refactored.

Otherwise, yeah, it's reasonable to have a stack of paper (so to speak) that are just paper to be sorted out. But becareful, you know what happens to a small stack of paper. :)

Rabbits.. I swear!

[ryantate]

I have a bunch of little modules in the My:: namespace. This avoids namespace collisions, makes it really obvious when I am using personal code (especially because I now explicitly import all subs) and makes it easy to find and move non-CPAN modules in the filesystem, for example when migrating to a new machine.

Examples: My::Uid for unique IDs (usually in a Webapp context); My::Logfile qw(/path/to/logfile) for easy debug logging; My::Security for easy password encoding (with a default random personal salt).

I'm confident there are probably CPAN solutions to these problems or, when there are not, I'm only saving myself a few keystrokes. I figure I can always map my own code (when it exists) into a wrapper around the Proper Module. And I certainly would never ship any code off to the world that included any of My:: modules -- I would simply find a substitute off CPAN or copy the code over.

[duelafn]

Like many, I use a module (docs, Dean::Util), however it also has the ability to remove dependency on itself from scripts and other modules. For example:

# In some script/module
use Dean::Util qw/map_pair nsign min_max/;
# other code ...

Then later to send the script to other machines:

perl -MDean::Util -we insert_Dean_Util_functions the_script.pl

This replaces the use Dean::Util line with the code for the subs requested (and optionally documentation). The insert_Dean_Util_functions function is a bit of a kludge, but it works.

This gives the benefits of using a module (old code uses improved functions when I get a chance to fix up code in the module) as well as the benefits of copy-paste (can transfer scripts to other machines without installing my module).