Developing a module, how do you do it ?

[mascip]

I have read lots recently, and read about lots of smart ways to create and manage modules, but i have never read the answer to a basic and primordial (to me) question :

Concretely, how to develop a module in a "handy" way ?

Until now i had all my files together in a directory, and they called each other with "lib". When i modified a file it was immediately taken into account by the others : it was easy.

Since i have started to try and develop a project with several modules, (created with Module::Starter::PBP for a first go), i don't need to use "lib" anymore, as i install my modules with "dmake install" and they go into my Perl directory.

But, it got more complicated and irritating for me to develop:
Each time i change a little thing in one of my modules, i have to type "dmake install" in the corresponding directory (thus having a terminal opened for each directory), so that the other modules will know about this change. It is very annoying.

What am i doing wrong ?

How do you develop your modules ?

Until you release your module "to others", do you keep on using "lib" so that your modules will know about each other?

I am discombobulated. Thank you for any piece of help :o)

[davido]

You have to work harder so that you can make it easier.

Version control: Put it under Git or something similar. That way you can easily create branches for various ideas or development tracks.

Your test suite can drive new features. As you come up with a new idea to add to the module, add a test for it. Be sure to also add tests that test the various units that contribute to the new feature, as well as testing the normal use case you anticipate. And don't forget edge cases.

make (or dmake), then you can prove: prove -b -v t/10newfeat.t. ...although I find that sometimes prove jumbles the chronology of stderr output and stdout output, so while developing I usually just, perl -Mblib t/10newfeat.t.

Here's a brief example of the steps:

1. New idea comes to you, and after doing the necessary research.... git checkout -b newidea <= This creates a new topic branch for your project.
2. Create a new group of tests: t/10newfeat.t <= It doesn't matter if these will end up being "finished product" within your test suite. Just code up a few tests that drive the unwritten solution how you would like to be able to use it. Mark the test(s) "TODO".
3. git commit -a -m "Created TODO tests to drive feature xyz."
4. Work on your new feature.
5. Anytime you want to see how your feature is progressing, either add another driver test to your test file, or invoke the existing ones: make; perl -Mblib t/10newfeat.t.
6. Whenever you get to a point where you wouldn't want to have to recreate work, or where you are about to take a gamble of some sort, don't forget to git commit -a -m "Milestone: xyz does abc."
7. Repeat the above coding, testing, committing steps as long as it takes.
8. Set a version number (possibly a Developer's version number).
9. Update 'Changes', MANIFEST.
10. git commit -a -m "Feature XYZ implemented."
11. git checkout master (or whatever branch you want to merge this new feature into).
12. git merge newidea (and iron out any merge conflicts).
13. make realclean, perl Makefile.PL, make, make distcheck, make test, make disttest, make install, then run your test suite against the installed version. <= All steps that will help to give you confidence that you didn't miss anything along the way.
14. Either tidy up some more, or if everything's ready to go: git tag -a -m "v0.12 Added xyz feature" "v0.12" 12asdf (the MD5 of your final commit).

Now you can push it to your external repo, upload to CPAN, move it into integration testing, go live... whatever you do next with your work.

Your steps and workflow will differ; collaboration, additional testing, keeping topic branches out of the master until after some other milestones are reached... whatever. And no two projects are alike, but that's the general notion of what seems to be my pattern.

[BrowserUk]

Somewhere in the Home Counties circa. 1930.

Lady Jane picks up the telephone receiver, "jiggles" -- she was assured that was the right term -- the contact "thingy" on the cradle -- she knew that wasn't the right term, but no one seemed to know what it was called -- a few times.

Placing the receiver close to her ear, she waited. Presently, she heard the unfamiliar, but renowned, shrill squealed greeting of the County telephonic operator -- the Estate operator really; but since the Estate encompassed pretty much the entire county, they very much were one and the same thing.

"Oper'ta. 'ho can I get for yur love?"

"Good morning. Mrs Patterson. And how are you today?"

she enquired. Civility & proprietary, hallmarks of her breeding, she thought.

"Oh M'lady. I'm ... I'm very well. Thank you M'lady. Sorry, When your line rang I assumed it would be Tollerson calling, I ..."

Cutting her off.

"No matter, Mrs Patterson. I wonder if you would be so kind as to put me in touch with my husbands office."

"Of course M'Lady. Right away. Won' be buta jiffy..."

She could hear frantic activity and indiscernible snippets of hushed conversation at the other end.

"Oh, and Mrs Patterson, since it is becoming the done thing to use the infernal contraptions oneself, and we are likely to be talking to each other on a more regular basis from now on, do drop the "M'lady". Call me Lady Jane. It it the 19030s all said and done".

"Of course M'l.. I mean Lady Jane. Thank you M ... Um Lady Jane. Would ya mind 'olding for a few seconds, I need to talk to the receptionist at t'other end.

"Certainly".

A few seconds went by.

"Okay Lady Jane, I'm putting you through to Miss Frobershire now. Go ahead caller."

"Hello", Lady Jane enquired tentatively.

"Hello, Lady Jane, Miss Frobershire here. How may I help you?"

"I wonder if you could convey a message to my husband for me?"

"Perhaps I could put you through to his secretary, Mrs Lyons?"

"Oh yes. That's a very good idea. Please do."

"Okay, please hold."

"Oh and .."

but the lne was dead. She waited and after a few moments she heard.

"Mrs Lyons, I have Sir John's wife, Lady Jane on the line for you."

"Hello? Lady Jane. Is something wrong?"

Enquired Mrs Lyons.

"No no. I just wanted to get a message to my husband. Nothing terribly urgent, but he told me that I really should make the effort to use this thing. We apparently pay quite a lot of money for the privilege of having it, and it sits here day and night mostly doing nothing."

"I see Lady Jane, well Sir John is in a meeting with the PM right now, I don't think that I should interrupt him if it isn't urgent."

"No, not urgent at all. Perhaps you could take a message and see he gets it?"

"Of course Lady Jane. How would you like the message to read?"

"Please ask him to swing by Fortnum & Masons and pick up a jar of their wonderful Potted Shrimps, only his mother is calling in for tea tomorrow and she does so love them."

"Is that all Lady Jane?"

"Yes... er. rather no. Could you also ask him not to be late home this evening as the Fothering-Smythes are coming for dinner and they are such a bore. I need him here to prevent me falling asleep in my soup course."

"Certainly Lady Jane. Is that all?"

"Yes. Thank you, Mrs Lyons. That's all. Goodbye."

"Goodbye Lady Jane".

The line went dead. "What do I do now?", she though. And settled upon replacing the receiver on its cradle.

Leap forward 80 years to present day.

Jane picked up her cellphone, swiped it unlocked and touched the contacts app. Swiped down until she found her boyfriends number, touched the txt icon and typed: "Pick up takeaway on your way home -- Chinese or Indian. XXX" and hit send.

"I probably should have abbreviated some of that, but it always takes me longer to decide how to abbreviate it than to type it in full.", she thought smiling to herself. "How old fashioned I am.". And proceeded to the next task on her to-do list.

---

We programmers have revolutionised and (in most cases) simplified the operating procedures of virtually every industry on earth. Except our own, which has gotten stuck somewhere in the 1970s, Same tools; same working procedures; same labour intensive, manual step 1; manual step 2; manual step 3, .... go to step 1; that we've being following for the last 30+ years. Except the list of steps has become longer.

With Perl, we got rid of two steps (compile & link), but then introduced six more by way of compensation.

There has to be a better way. There *is* a better way -- and no, I'm not talking about GUI-IDEs with code wizards. But certainly some of the functionality of the better ones would be a good start.

• When I save a source file, it gets syntax checked immediately and automatically and moves the cursor in my editor directly to the line of the first error, if there are any.
• When the compile is clean, it puts it straight into the CVS-equivalent and then invokes the test suite.
• If a test fails, the test suite stops, and the appropriate test file is loaded directly into my editor at the failing line with the failing output displayed nearby.
• When the test suite passes, it installs the module automatically into the development library.
• We can now switch to the source buffer of the program that uses it and try it out.

And pigs will fly, world hunger will end and all will be right with the world :)

[Anonymous Monk]

There *is* a better way...

How'd you do that?

[sundialsvc4]

One of your best.   “++”

[BrowserUk]

You have to work harder so that you can make it easier.

That's easier? Ug! What a palava. :)

[Anonymous Monk]

"You have to work harder so that you can make it easier."

Pretending to be clever (palava) while simultaneously failing to read the spirit of the language is what keeps you in robot mode, robot boy. What davido surely meant was to merely strive to make your life easier. To focus efforts in a valuable way.

You use too many words, BTW. Here's yer gold star.

[grantm]

I'm wondering if you're using regression testing. The fact that you need to install stuff regularly suggests that perhaps you're doing that to manually test each small change. My workflow is to make changes to tests and code in parallel and regularly run prove -l or make test (or dzil test for code that I package with Dist::Zilla). When I'm happy that I have the required features working, then I build a package and install it. If you're not familiar with Perl's regression testing tools, this document might help.

Also I'm sure you realise this, but just in case you don't ... a distribution can contain more than one module. So you might use Module::Starter to generate a skeleton distribution including a .pm file under the lib directory. But you can put other (related) modules in the same lib directory so when you develop your code and your tests, the family of modules are developed together. Then a make install will install them all. To put it another way: if you want to make 5 related modules you don't need to run Module::Starter 5 times.

You might also get some mileage out of setting the PERL5LIB environment variable to point to the lib directory containing the code you're currently working on. Then when you run a script it will use the development version of anything it finds in that directory and the installed version of everything else.

[tobyink]

Agreed -- if you're working on a bunch of tightly coupled modules, they should live in the same source tree (thus the same "lib" directory), so they all get tested, published, upgraded as a clan.

If your modules are loosely coupled enough to live in their own source trees, then a good test suite should prevent regressions in one affecting another.

The exception to the above is where one module uses a feature of another which has no test cases. So make sure your test suite is comprehensive enough to cover everything.

Case in point: I recently contributed a feature to XML::LibXML which used overloading on XML::LibXML::Element. However, once 1.91 (and 1.92, which was a quick bug fix for Perl 5.8). This broke the ability to do:

if ($element1 == $element2) { ... }

Why? Because if you overload one operator, you need to overload every other operator you plan on using.

This bug was fixed very quickly in XML::LibXML 1.93, but how did a release with this bug actually get published? Simple: the ability to check equality between XML elements was never checked in the test suite, so even with this regression the full test suite was passing.

The moral of this story is: make sure that your test suite covers every public interface of your module that its users might be expected to use.

[JavaFan]

Until you release your module "to others", do you keep on using "lib" so that your modules will know about each other?

No. I use PERL5LIB, just like God^WLarry intended.

I only need to set PERL5LIB once (in my .bashrc), and I don't have to set anything in any Perl file.

[tobyink]

By the way, forgive me if this is obvious, but you do know about Perl's "-I" command-line option, don't you?

[mascip]

Thanks a lot to all of you !
I am really just starting to 'learn becoming an "efficient" programmer' (i have started almost from scratch), and your help is very appreciated.

Your messages made me realize a bit more that each person and project can/should have a workflow that suits it.
It can be the same as other persons/project, for the sake of compatibility or easier communication. Or it can be personal and unique for the sake of efficiency, if there is no need for communication and work protocols.

I work alone on my code (it might change some day - who knows), so i understand BrowserUk's efficiency motivations. And i would like to be able to work with others some day, so i want to try and learn a bit of Revision Control and "Test::* testing" for example.

In your messages i have learned many things :

- That i don't need to create 5 different Module::Starter directories for my 5 modules, and that makes quite a big difference in terms "lib" thingies.

- That i could use

perl -I

I didn't know about it, thank you. Are PERL5LIB and lib not easier to use? To me, the 3 of them seem to do exactly the same : give the information of "where are these sources that i need". PERL5LIB does it through an environment variable, lib does it in each source file, and -I does it on the command line. Any other difference?

- An alternative idea for having "non-boolean tests" directly in the source. What i don't like about it, is that it makes the source less readable : it becomes harder to search the interesting parts of the code.
But i like that it is run each time i run my code, and that i can be more creative than "ok". But...i like "ok" because it is easy to share : everybody understands what it means, when it appears on the screen.

- A clear tutorial on Revision Control

- Easier ways to manage my tests and run them, and a text about Regression Testing that i will most probably read.

... and a nice story !

Now i know how i want to start developing my own modules. Until now i felt lost in the profusion of tools and techniques, and by tiny details which make big differences (my 5 Module::Starter directories where a real pain).

Thank you, you're all great!

[BrowserUk]

An alternative idea for having "non-boolean tests" directly in the source. What i don't like about it, is that it makes the source less readable : it becomes harder to search the interesting parts of the code.

I think you got the wrong end of the stick about this.

In the normal way of things, a module, Your::Thing.pm, looks like this:

package Your::Thing;

## The subroutines and stuff that implement your module

1; ## finish with a true value to satisfy require/use

And the tests live in a bunch of .t files.

The "alternative idea" is that Your::Thing.pm looks like this:

package Your::Thing;

## The subroutines and stuff that implement your module

return 1 if caller; ## return a true value and exit when loaded as a m
+odule
package main;

### Write tests that use Your::Thing here.

...

You use the module in the normal way via use or require, and it works exactly as with the 'usual way' above.

To run the tests, you type:

>perl \Perl\site\lib\Your::Thing.pm

And the code that comes after the return 1 if caller; statement gets run. It really is as simple as that.

And whilst the tests and code exist in the same file, the module code is above that line, and the tests are below it. There is no interleaving. Nothing is "less readable". Nothing is "harder to search" for.

Indeed, maintenance is greatly simplified by having the code and tests within the same file.

Try it. You'll never go back :)

[tobyink]

To me, the 3 of them seem to do exactly the same : give the information of "where are these sources that i need". PERL5LIB does it through an environment variable, lib does it in each source file, and -I does it on the command line. Any other difference?

Ultimately there's only one way of telling Perl where to search for modules: the global @INC array - it's a list of directories that Perl uses to search for libraries. PERL5LIB and "-I" are just ways of informing Perl to pre-populate @INC with certain directories.

And use lib just manipulates @INC while your code is being compiled. (You can look at lib.pm and see what's going on - there's nothing especially mysterious.)

They're each useful in different ways.

If you have some paths that you always (or nearly always) want Perl to search in, then use the environment variable. You can set it in your .bashrc/.tcshrc or some other script that is executed as soon as you log in, and forget all about it.

use lib is useful if you want to hard-code library paths that will be searched for specific scripts. It's not especially good for code you hope to publish for other people to install and use though, because they might plan on installing modules somewhere else. Relative paths can also be problematic with use lib because they are treated relative to the directory Perl was executed in, not relative to the file in which the use lib appears. (There are modules like "lib::abs" which improve the situation.)

The -I argument for perl is useful for most other situations. It tends to be what I use when I'm testing Perl modules I've written and intend to publish. I'll use a directory structure like this:

  p5-my-module/
        lib/
              My/
                    Module.pm
        examples/
              demo.pl
        t/
              00basic.t
              01advanced.t
        xt/
              01pod.t
              02pod_coverage.t

Then I'll have my terminal mostly in p5-my-module, and I can run things like:

perl -Ilib t/00basic.t
perl -Ilib examples/demo.pl

I find that use lib can be handy used within test suites to help the test cases find additional testing modules (which might be inside "p5-my-module/t/lib/"). This is because although use lib isn't great for stuff that's getting installed on other people's machines, test suites don't get installed, and they are run in a very predictable way.