Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Use modules or roll your own?

by kvale (Monsignor)
on Jul 29, 2002 at 05:23 UTC ( #185888=perlmeditation: print w/replies, xml ) Need Help??

Recently a Seeker ran into some trouble with an example script from a popular Perl textbook, and the immediate response from a prominent monk was "Don't reinvent the wheel! Use module X instead!" This surprised me, as I thought learning how the script worked would be a worthy goal.

But the conflict in opinion brings up a larger question: at what level of program detail is it best hand off duties to a black box, i.e., a Perl module? And if a module is used, what sort of effort should one make to understand its innards?

One of the greatest strengths of Perl is the massive CPAN library of modules available. Code reuse really works here and allows me to be Lazy, in the good sense of not wasting time duplicating fine work already out there and Impatient, in the sense that I can write powerful programs quickly.

But at the same time there is a danger of not really learning about the problem I am trying to solve. I might learn just enough to realize that there is a module that can solve part of my problem. After downloading the module, I might skim just enough of the synopsis to cobble together some code. If it works, great! But if not, I am in trouble - I don't know enough of the underlying structure of the problem or the module to figure out if the bug is in my driver program, my interface code, or the module. Worse, if it sorta works, I might be tempted to not to bother trying to figure things out, because, hey, I've got more important things to do than noodle about in the details. This is laziness and impatience at its worst, and can lead to Cargo cult programming.

For instance, if one was to implement a multitasking server there are several possible approaches:

  1. Use Parallel::ForkManager as is
  2. Copy code for a forking server from the Perl Cookbook and modify it to suit the task
  3. Learn about  fork() and  exec() from the Perl and Unix manpages. Implement the server from scratch.
Which would you choose?


Replies are listed 'Best First'.
Re: Use modules or roll your own?
by Abigail-II (Bishop) on Jul 29, 2002 at 10:04 UTC
    If you already have decided that you are using Perl, which is already doing a lot under the carpet, the step to using a module isn't that big.

    IMO, a decision to use a module depends more on the module than on the philosophical question of using it module itself. If a module has a bad API (like pointlessly using OO), a difficult manual page or many dependencies, I might opt to role my own.


      Im kind of curious as to what you would consider "pointlessly using OO". Care to expand on that with some examples?


      Yves / DeMerphq
      Writing a good benchmark isnt as easy as it might look.

        use Lingua::EN::Numbers; my $n = new Lingua::EN::Numbers; $n -> parse (1281); print $n -> get_string, " fish in the sea.\n";
        Which is the reason I wrote Lingua::EN::Numbers::Easy where you can do:
        use Lingua::EN::Numbers::Easy; print "$N{1281} fish in the sea.\n";
        to do the same.


Re: Use modules or roll your own?
by rozallin (Curate) on Jul 29, 2002 at 11:43 UTC
    There is nothing "wrong" with using any of those three approaches. I think the end results from each path would be pretty much the same. The difference lies in the experience gained by the programmer.

    As an initiate to All Things Perl, I am indeed very tempted to use modules. It's a very alluring trap for the inexperienced to fall into. But in my humble opinion, modules should only be used when they fit the problem exactly or when you need to write a program quickly and aren't too concerned about roughness round the edges.

    I very rarely write things from scratch, except if I can gain some profound enlightenment from it. I feel it's better to meditate on some existing code and then modify it to your particular needs. I find that it's better that way, because you learn how to program well from good programmers, and learn how to spot "bad programming". There's a chance that, even if you are quite competent at Perl, you could be slightly misguided when writing from scratch. But then again, your mileage may vary.

    -- Rozallin

Re: Use modules or roll your own?
by belg4mit (Prior) on Jul 29, 2002 at 06:36 UTC
    I think the concerns about not learning by using modules etc. are not answerable. Whether or not somebody chooses to learn by peeking behind the curtain, or fix every last thing is dependent on the metal they are cast from (or if the metal's been worked enough). Basically, don't worry be happy.

    I'd roll my own, checking perlipc if I had to since it's been awhile. If it was extra fancy I might look at POE.

    perl -pew "s/\b;([mnst])/'$1/g"

      I guess the author of the original post has got a legitimate issue to worry about. His concern seems to be whether reusing the old wheel will work in his particular case to solve a 'custom' problem. This happened to me just recently. In one of my projects, I set out to write a bunch of modules to do RDBMS interaction. However, after realizing that I'm essentially doing the same work that Class::DBI already does, I hastily switched.

      Later, however, I've discovered a number of features that the module lacked and that was a real set back as I was hoping to meet tight deadlines. So, I ended up hacking into the module to fix a few things (such as deleting cross reference records from tables with multiple primary keys). On one occassion, the behavior of the module was so unintuitive I ended up messing my database data in unexpected places.

      In all, I say, if you ever go to reuse an old wheel, please do study it closely first. Check for any possible defects or lacking features before use. At times it may even be hazardous to introduce an unknown or poorly studied 'formula' (aka module) into your equation (aka project).

      # Under Construction
Re: Use modules or roll your own?
by Tomte (Priest) on Jul 29, 2002 at 09:22 UTC
    I don't think your more general question (that is, the node-title :) has a final answer, and the more specific question ("Which would you choose?") neither.

    In my case, the answer depends: private or job project?

    • private: roll my own, if interesting enough
    • job: use a Module if it's feasible, otherwise answer2, including copying from Modules.

    In my experience, reusing code from other, more experienced coders sooner or later leads you to better understand the problem-domain(s) you're acting on, as well as the language of the day, while saving a great amount of time between project start and delivery/deployment

    after all, I'm a coder because I'm lazy :-)
Re: Use modules or roll your own?
by FoxtrotUniform (Prior) on Jul 29, 2002 at 14:43 UTC

    As always, It Depends.

    If I have plenty of time on my hands, and I'm interested in learning about fork/exec, then sure, I'll roll my own. I'm in this game to solve interesting problems, even if they've already been solved a billion times.

    Otherwise, it depends on what modules are out there. If there's one that fits my needs (doesn't have a horribly broken interface or implementation, has all the features I want, doesn't make assumptions that aren't valid for my project, etc), I'll use it. If one of the existing modules is "close enough" to what I want, I'll patch it. Otherwise I'm pretty much stuck writing code from scratch.

    In this case, since I've already been introduced to fork(2) and friends, I'd probably use an existing module.

    The hell with paco, vote for Erudil!

Re: Use modules or roll your own?
by chromatic (Archbishop) on Jul 29, 2002 at 14:45 UTC
    Worse, if it sorta works, I might be tempted to not to bother trying to figure things out, because, hey, I've got more important things to do than noodle about in the details.

    Hmm, I'd argue that most of the people I know who claimed that rolling their own modules was a "learning experience" fell victim to the same problem. I find this argument unconvincing.

      Not only that, but where do you stop?

      Roll your own pragmata?
      Roll your own perl?
      Roll your own libc?
      Roll your own device drivers?
      Roll your own OS?
      Roll your own hardware?


        Roll your own perl?

        Roll your own awk (like Larry Wall did)?

        Roll your own OS?

        Roll your own unix (like Linus Torvalds and/or Richard Stallman did)?

        Roll your own gopher (like Tim Berners-Lee did)?

        Seems like we have a lot of respect for those who chose to reinvent the wheel and got something good.

        mitd notes that it may be possible at the quantum
        level to 'roll your own universe'. In which
        case Abigail's list could only be
        achieved through reverse engineering.

        mitd-Made in the Dark
        'Interactive! Paper tape is interactive!
        If you don't believe me I can show you my paper cut scars!'

Re: Use modules or roll your own?
by Maclir (Curate) on Jul 29, 2002 at 14:49 UTC
    I would disagree with calling a Perl module a "black box". Maybe a "dark box", "light grey box", or a "cream box", depending on one's knowledge of Perl. Remember, with all Perl code the source is available for you to examine - and even modify if you wish. For someone wishing to learn about a new aspect of Perl, seeing how one welliimplemented solution approached a problem is an excellent way to learn.

      Oh yeah! If I had a dime for every new concept or nifty trick I learned from poking around in someone else's module code, well I'd be eatin' steak instead of ramen.... 8)


A beginner's perspective
by BorgCopyeditor (Friar) on Jul 29, 2002 at 18:29 UTC

    I've been writing perl for about 2 months. It's the first programming language I've had anything to do with since I learned BASIC on my TRS-80 as a kid. (Unless we count Javascript, but there I was mostly cutting and pasting to put some doo-dads on my web pages, something that has lost its charm.)

    Since I knew HTML pretty well and had some specific needs (e.g., character encoding), I "rolled my own" instead of learning The fact is, I didn't know I was supposed to do any different. The few example scripts I happened to come across all printed HTML directly instead of generating it via a module.

    Coming (finally) to the point. Because I was unaware of the module, I came to learn a bunch of neat things on my own. Trying to get perl to parse other people's pages was hellish, as was trying to print output in a variety of encodings (based on user configuration), but it taught me a bit about regular expressions that I might not have learned otherwise. And learning how to parse forms led me to investigate pack.

    In short, I approached things as opportunities for exploration as much as tasks that needed to be done. (Since this is a hobby, I'm lucky to have that leisure.) Eventually, discussions here and on usenet (where I went when I couldn't figure things out for myself) led me to rethink how I had been coding certain things. But I'm not sure it would have been better to be exposed to this sage advice first, or be handed a module, because I might have missed a good opportunity to exercise my wits on a problem.

    --Your punctuation skills are insufficient!

Re: Use modules or roll your own?
by Popcorn Dave (Abbot) on Jul 29, 2002 at 15:44 UTC
    Personally, for your question I'd do 2 and 3.

    Myself, I prefer to try to "reinvent the wheel" so that I gain a better understanding overall of Perl. Then once I know what I'm doing, if there's a module written to do the same thing and it produced less pages of code I'd switch to the module.

    Argueably, there are things that a beginner in perl could perhaps not do on their own without a module, but I would say that if they're trying to do something that complex, perhaps it's a bit beyond their scope at the moment and they need to step back juts a little.

    Some people fall from grace. I prefer a running start...

      If you want to have a better understanding of Perl, you shouldn't be rolling your own modules, but roll your own core functions.

      Write your own print(), qr() and tie().


        Well as I'm still learning Perl -- not that you ever stop learning -- I personally tend to try to do things without modules first so that I get a better core understanding of Perl.

        As far as writing my own print, etc... if I was interested in doing that for a project I was working on, I probably would give it a shot. Until that point though, I'll continue on down the road I travel now.

        I'm currently writing a program that parses HTML pages, and yes I know I could use a module, but for me, I'm sharpening my regex skills the way I'm doing it. When I finish it, I will look at using the module to see if that simplifies my code, but not until.

        That's just me perhaps, but it works for me.

        Some people fall from grace. I prefer a running start...

Re: Use modules or roll your own?
by mstone (Deacon) on Jul 30, 2002 at 03:01 UTC

    There is no Right answer. There are only tradeoffs.

    Rolling your own means you have to explore the problem space. On the plus side, you end up with a good working knowledge of what you're trying to do -- what's possible, and what's not. On the minus side, it takes time to acquire that knowledge, and if you try to do it entirely on your own, you face an absolute mountain of special-case testing before you can truly say that you understand the space.

    Using modules means you have to trust someone else to explore the problem space for you. On the plus side, it's fast, comparatively easy, and for mature modules, saves you a whole lot of 'whoops' time. On the minus side, you have no guarantee that the person who wrote the module was actually trying to solve the same problem you were, and you might not understand the problem space well enough to make that call.

    To decide which option to use, look at those conditions, and decide which one offers the best benefit/risk ratio.

    Core functions like printf() are so utterly common that you can be reasonably sure the canned version will do everything you want it to do. Well-defined operations, like sorting or math functions, are narrowly enough defined that they're easy to test, and are also likely to make good modules. Extremely sensitive systems, like hard real time controllers, cryptographically-secure random number generators, and concurrent transaction managers, are best left to the experts. In general, if you can specify the problem to an excruciating level of detail, you can probably find a module that already does what you want, and you'll know that it does what you want.

    The looser your problem specification, the less likely you are to be sure that a specific module does exactly what you want. Should you use to generate HTML, or should you abandon inlined formatting entirely and go with a template-based system? If you choose to use templates, should you use HTML::Template, or do you want to do something the module doesn't support? (one of my template-based systems, for instance, emulates lambda calculus -- you can replace one marker with text that contains another marker, and the system will keep grinding away until everything has been resolved. it also uses a hierarchy of dictionaries that map markers to replacement text) Just using a module is no substitute for knowing what problem you actually want to solve.

    Personally, I think the only time you're really safe deciding between modules and hand-rolled code is when you could write the module for yourself, from scratch. Until then, you take your chances. Decide which option makes you most comfortable, and do that.

    If you do choose to use a module, ask yourself why you shouldn't open the source files, work through the code, and learn what the module does. It's a great way to explore the problem space without spending so much time waiting for some random glitch to expose a hole in your basic assumptions. If your answer is, "because I really don't care that much," that's fine. Just remember that when you find yourself being tempted to get religious about using that module. Getting dogmatic about something you can't be bothered to understand is pretty much the definition of cargo-cult programming.

    If you don't use a module, ask yourself why you shouldn't look at other modules that do roughly the same thing, and use them as references. If your answer is "because I really don't care that much," that's fine. Just don't get grumpy when you ask questions and someone says "RTFS."

Re: Use modules or roll your own?
by Steve_p (Priest) on Jul 29, 2002 at 17:11 UTC

    I work in an environment where developing new code is easier than attempting to get a CPAN module installed. In this case, it is often easier to reinvent the wheel in the way I need it, and install the code in my own modules, than spend months try to get "unsupported" code installed.

    Often I have to do it two ways. I can experiment at home all I want with the CPAN modules, while I dig throught its guts trying to figure it out. At work, I have to reimplement the idea for my specific task so as not to infringe on any licenses.

      I work in an environment where developing new code is easier than attempting to get a CPAN module installed.

      Wrong. If you can write a script and run it, you can install a Perl module for use with that script. There are at least two options:

      • install the module tree in the same directory with your script and use lib '.';use Module::Name;
      • for pure Perl, download the module, copy and paste the relevant bits into your script (advantage of retaining appearance of single script)
        I know I can do these things, however, doing these things are not possible given the environment I work in. Also, if I copy and paste from a module to a script, I will likely violate the license of the modules.
Re: Use modules or roll your own?
by Aristotle (Chancellor) on Jul 29, 2002 at 22:13 UTC

    I've written my thoughts on the topic in another thread relatively recently. Basically, do play around and roll your own to learn.

    When it comes to production code however, you should use an existing module that offers what you need. Modules are under the scrutiny of a larger audience than your own code. Also, using a module reduces the amount of both code and documentation that you have to write as well as maintain. You better have a really, really good reason to sidestep these benefits.

    I did write my own CGI GET/POST parser; my own templating system; my own recursive directory tree descent. All of these were as bad as any other newbie's stab at them would be. Some of their deficiencies became obvious to me soon while others I didn't have the experience to think of and had to read about. It was indeed a learning experience I do not regret; but with that experience under my belt, I also wouldn't nowadays even dream of rolling my own for these tasks.

    Makeshifts last the longest.

Re: Use modules or roll your own?
by Maclir (Curate) on Jul 29, 2002 at 22:48 UTC
    Some more thoughts on this - and let me put my answers into context.

    Context 1 - You are working for me and being paid to produce high quality applications that are maintainable, efficient, and developed within a reasonable time. If I catch you "rolling your own code" when I know there are perfectly good and widely accepted modules that you could (should?) have used instead, you better have a damn good answer. I am not paying you to be creative, or to learn everything there is to know about forking (or CGI or threads or . . . ).

    Context 2 - You want to increase your understanding of Perl and related matters on your own time. Delve into the deepest innards of whatever modules take your fancy. Try to improve them. Discuss your findings with the module author(s).

    Context 3 - You are undertaking a programming course at university of college. Then this depends on the purpose of the assignment I had set you. If it was to show your understanding of CGI request processing, paramater handling and so on - then I would want to see your own code (and not something ripped off from If it was an assignment to develop a working, live web application, then makes woudl be deducted for not using, CGI::Application or whatever modules made sense.

    I suspect the initial case that started this thread fits into context 2. So, learn from the masters - and improve on them if you can. Wasn't there some discussion several months ago about Isaac Newton being able to see further that others because he stood on the shoulders of giants? In CPAN there are many giants whose shoulders we can all climb on. Some are true collossus, while others are barely taller than you and I.

Re: Use modules or roll your own?
by vek (Prior) on Jul 29, 2002 at 19:17 UTC
    I don't know, sometimes having a basic understanding of what's going on "under the hood" instead of being sheltered by an API isn't necessarily a bad thing. At the very least when you're having a conversation you can at least answer a question instead of saying "uh, I've no idea, all I know is that I just call foo () in my program and stuff just happens." :-)

    -- vek --
The Third Way
by crenz (Priest) on Aug 01, 2002 at 14:52 UTC

    So you find that the existing modules don't do things the way you want it. Is starting from scratch really the only way to go? I beg to differ. Often, you'll find that a module is close to what you want, even though it is not your style. Why not make some changes so it fits your purpose and submit the changes to the author? This way, you will be able to

    • Save time (even if you end up only using half of the existing module).
    • Learn from other people's code (because you have to understand their module first.)
    • Learn to be a collaborating hacker instead of a Lone-Ranger-Coder.
    • Get what you want, and in your style (at least partly, in those lines that you code).
    • Have less "fame", because the module is not published under your name. So what.

    True, there are opportunities when you need to reinvent the wheel. Linus decided he needs another OS -- but he was clever enough to make use of the GNU utilities instead of doing everything himself. Linux, emacs, perl, apache -- all these are good examples. On the other hand, there are for example (too) many modules for command-line parameter processing on CPAN. Probably not all of them are necessary...

    If you feel you can really learn something valuable from writing the code from scratch instead of just understanding other people's code -- go for it. Just don't expect that everybody else will be enthusiastic about it. If your approach is radically different and helpful -- go for it. If not, consider improving your skills by trying something new. Try to be more lazy and don't do things you don't have to do.

    I guess at the end of the day, it all boils down to

    use wisdom;
Re: Use modules or roll your own?
by japhif (Scribe) on Jul 30, 2002 at 02:46 UTC
    The answer to this question depends greatly on the situation. You mention the Seeker was reading a Perl Textbook. The answer depends on why he or she was reading the text book. Was it to learn a new trick for work or school? Or was it to learn a required skill to complete a project that's due in 1 or 2 weeks? I used to be an advocate of reinventing the wheel, but I've backed off some (after reinventing the wheel where I shouldn't have). If the purpose in why you're reinventing the wheel is educational then go for it. After you reinvent the wheel, see what other people's version of the wheel looks like. Then try it again with new ideas that you've learned. It is a time consuming process but a good way to find the road to mastery. If, however, you're reinventing the wheel to use in production code, you have to be careful, very careful. If you insist at that point on reinventing the wheel then do it, but remember you have to pay your dues. If you factor in testing, testing, and more testing in your version of the wheel then it seems like reinventing the wheel may not be such a great idea (most people hate testing, but it is a crucial step in writing good code). But if there are not existing modules to use that fit your exact needs, you may have no choice. But if you do have a choice and a deadline in the near future, think twice before reinventing the wheel.
Re: Use modules or roll your own?
by feloniousMonk (Pilgrim) on Jul 30, 2002 at 15:59 UTC
    Personally, I use modules whenever I can if the following conditions are met:

    1. Deadline is short (i.e., I have a deadline :-) )
    2. I know the module is quite robust, maybe overkill but useful for the problem at hand
    3. I have 'bigger fish to fry', i.e. the module solves a problem so I can concentrate on the 'main issue'
    4. And sometimes, I don't know an efficient way of solving the problem but someone else already has

    In case #4 the module route just gets things working then I can go back
    and actually learn what I need to in order to solve the problem myself

    -felonious --
Re: Use modules or roll your own?
by valdez (Monsignor) on Jul 30, 2002 at 09:26 UTC

    In some cases it is not a bad idea to roll your own modules, if it can helps you understand what you are trying to achieve.
    I tried many times, especially with templating systems, and the final result is that now I'm able to choose the best approach/solution for my needs.
    Of course, my first visit every morning is CPAN :)

    I'm sorry I can't use more than one vote on the node of mstone

    Ciao, Valerio

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://185888]
Approved by Kanji
Front-paged by rob_au
[Corion]: marinersk: Well, I have done select statements like select sum(foo) as "Total Amount", ..., but to have a table like that makes me shudder
[Corion]: SuicideJunkie: :-D
[marinersk]: SuicideJunkie LOL
[choroba]: Woohoo! Fixed a test that hasn't run for 3 years.
[marinersk]: Corion Yes, sometimes whitespace in column headers is acceptable, but I still consider it be less than desireable if that query might get revectored for an ETL-esque process...
[marinersk]: choroba++
[choroba]: it's a long running test, so it's normally skipped unless an env var is set
[choroba]: nobody has been bothered to set the variable in the last 3 years
[marinersk]: sub newtest{my $expected_result = &target('foo'); my $actual_result = &target('foo'); if ($actual_result eq $expected_result) { &tdd_success(); } else { &tdd_fail(); } } # Test works after three years!
[choroba]: or nobody bothered...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (11)
As of 2017-05-25 15:08 GMT
Find Nodes?
    Voting Booth?