RFC: Perl Testing -- How to Introduce to a team

[NovMonk]

Wise monks,

My company is facing a transition. We have been writing perl scripts-- little snippets of code to get certain text outputs-- or modifying longer programs (adding logic to affect printed text output). We have been using shell scripts to drive our processes-- the most ungodly mess you can imagine, and which varies from job to job based on who set it up, and how many years ago. The good news: we have recently purchased some new software and we are retooling our existing processes to use it, and moving from flat file to database (MySQL) data.

As we talked about designing this new framework, I timidly suggested adding testing modules to the process Up Front, and as a reward, I was tasked with researching testing, and training our developers to do it. The closest any of us have been to testing is adding print statements to code while debugging-- one of us (not me) actually uses the Perl debugger with some facility.

It's not a matter of convincing them we need to be testing-- everybody's sold on that. We are all just getting overwhelmed with where to start. I've read the posts on testing here found through Super Search, and I have printed some out to share. I've gone to CPAN and printed off several of the modules (Test::Simple, Test::More, the tutorial on testing, etc). I've been reading books on testing and software construction in general: Perl Medic, Perl Debugged, Perl testing: A Developer's Notebook , and Code Complete.

One problem is: most of us are self taught, or have only the haziest ideas about Object Oriented Programming, or writing/ using modules at all. Some things in our process are obvious- you want to test that needed files are where you expect them to be, are in the right format, and that the program generates the expected outfiles and sends them where they need to go. But-- What else? If you’re supposed to design tests before you code, what else should we be looking for?

My plan for training is to pass out some of the CPAN material on Test::Simple and Test::More, and the posts from here, and parts of the books, some of which we are buying as a company. I’m also going to touch on Perl Tidy and Perl Critic, as ways for us to enforce these coding standards we’re agreeing on. What I’m looking for from you guys is other suggestions, things I might think about, ways you’ve presented testing to programmers you’ve mentored over the years. Things you wish you’d known when you were starting out. Testing procedures you inherited when you started with a company, that you wish you could change.

We’ve this fantastic opportunity to design procedures and policies that will make our lives easier, make us better programmers, and make our company, and us, lots of money. And that is also a little overwhelming.

Many thanks in advance,

NovMonk

[philcrow]

The single best thing I learned about testing from Schwern is to make sure that every time someone finds a bug, they write a test to show the bug before they fix it. This works at the beginning of a project if, as you add feature requests to your list, you write tests for them as you write the code to implement them.

Hence, don't get bogged down in thinking about every possible thing you could test, just get started writing some tests and write more for every missing feature or bug.

Phil

[NovMonk]

I've been wondering about that--in such a case, do you write a test For the bug (passes if it hits) or For the Solution to the bug? I'm thinking the former, and then just Reverse it when it's fixed (passes when the fixed code runs right.)

Thanks for the comment.
NovMonk

[philcrow]

I write the test so it fails when the bug exists and keeps failing until I fix it. If you don't want it to show up for some period of time, mark it TODO.

Phil

[imp]

Congratulations. Writing tests for my modules is one of my favorite parts of development, as I recently commented.

Between the books you listed and the perlmonks nodes you have likely read there isn't much that I would add to this node about testing, so I'll focus on topics that reduce the number of failing tests.

When writing code you will frequently solve the same problem multiple times. This can be reduced by reusing modules from CPAN, and by learning to recognize those patterns and creating resuable components for that design pattern.


• Design Patterns: Elements of Reusable Object-Oriented Software
• perldesignpatterns.com

Investing some time in improving your programming practices would be good too. The following should be required reading for professional perl programmers:

• The Pragmatic Programmer: From Journeyman to Master
• Perl Best Practices

When preparing your own tests it would be a good idea to look at what other people have tested in their projects. Choose a couple CPAN modules that you use regularly and look over the test scripts that they use. To do this download the .tgz distribution, untar it, and look in the t directory.

[NovMonk]

Thanks for these great ideas. Seeing other tests on similar or familiar code Would be very helpful. I'll try that.

And your node was one I read and printed off. I'll have to take a look at these other books, too, and that website. Problem with any books, of course, is that we're starting to code Now, and none of us has time to digest what we really need to to be prepared. But that's pretty much the way of things-- we're never prepared for all we have to do to get through life. Ah well.

Thanks for the suggestions.
NovMonk

[xdg]

If you're testing already-written code, I'd do it top-down, testing overall results first, then working to more granular levels. If you're writing new code, I'd do it bottom-up, module-by-module, function-by-function.

Here are some posts that I've bookmarked that you might find useful:

• Wanted, more simple tutorials on testing
• Test driven development and glue code
• Functional and Unit Test Redundancy
• Application Testing: Custom Module or Perl Test Files?

[ptum]

As with documentation or the use of source control, this is a case where you need to measure and report on the behavior you want to incentivize.

I once worked for a group that was required to measure the way they spent their time, and most of the group members were 3-6 months behind on reporting. I was tasked with getting everyone 'current' -- a thankless job requiring some rather Orwellian adjustments of history. I hit upon the happy idea of offering gold, silver, green and blue stars for those who complied with the requirement to various degrees, and published the names (and awards) in our monthly newsletter. It was a silly thing, but people actually took a childlike glee in getting more stars than the guy in the next cube.

Now I measure POD coverage and compliance with our Subversion repository, and I have a little report on our departmental webserver that shows where everyone stands with respect to those metrics ... at the end of each quarter we compete with one another to see who can get a better data point on the graph. If I were in your shoes, I would do a similar thing with test coverage.

Most of us didn't get enough gold stars in grade school ... don't underestimate this powerful (and lighthearted) motivational tool.

[badaiaqrandista]

It seems that my company is in the same phase as yours, but we've only got 2 programmers and are overwhelmed with projects. I've been trying to implement tests for the existing code base (most of it is nicely modularized) whenever I have spare time. It looks that the thing that will work is to start from the simplest thing you could test.

It took me weeks to try to come up with good testing infrastructure and then I realize that I should've just started it with whatever I've got and go from there.

For example, when you are testing scripts, you probably want to test whether each arguments will give you correct behaviour. The reason is because that's what you currently have and to be able to test each subroutine in the scripts requires you to modularize the script and put those subroutines into a module, and that's another hurdle. That could demotivate you from test.

TDD is good, but without enough testing experience you'd gain with simple tests like that, it is hard to be able to express your design in terms of tests. Frankly, I've never use TDD, but it seems that to do it properly, you need experience in testing existing code.

If you are still learning how to create module, you can also learn that from your tests. The idea is: start from testing the script then work your way to tests every single branch of your code, xgd called it top-down in the other comment. You can't test each subroutine in a script, unless you split that into modules. After you move them into modules, they'll be easier to test. Then you want to test the bits inside the module, you factor that bit out to another subroutine. So doing this gives you insight of how to create modular module.

So, the conclusion is: just do it. Training can only explain the modules you can use to test and how to use them, but to actually understand how to test, each member in your team must start doing it.

[Herkum]

I think that the most important testing module that I learned was Test::Class. It is basically providing a OO structure to writing your tests. A good example of this is writing data Models that are working with a datbase.

I use the built-in setup and teardown methods to declare each set of tests as a transaction. When I do this I don't have to worry about which tests were run first. All db changes done for a test are automatically rolled back when they done.

[adrianh]

I think that the most important testing module that I learned was Test::Class. It is basically providing a OO structure to writing your tests.

While I am (unsurprisingly) quite fond of Test::Class, I'm not sure I'd recommend it as a first step if you've not done testing in Perl before. Especially if you're not particularly familiar with OO.

Start with Test::More and friends then, if you start finding you're wanting to factor out common testing functionality, take a look at Test::Class.

[Herkum]

It does have a higher learning curve than just using Test::More. The problem with avoiding an OO oriented approach is that they will quickly run into issues once their testing needs become significantly detailed. Better that they tackle good practices right off the bat and avoid reenforcing procedural programming that they are used too.

[jkeenan1]

Between the research you discussed in your OP and the comments from other Monks, you have more than enough information at your disposal to begin to test the Perl way. But I want to call your attention to the fact that you described at least three separate problems in your OP -- problems that require additional reflection to solve.

The first problem is how to write tests with Perl. The other Monks have provided good advise, to which I would only add that, at a more advanced stage, learn how to conduct coverage analysis with Devel::Cover.

Second, "most of us are self taught, or have only the haziest ideas about Object Oriented Programming, or writing/ using modules at all." Note that this is a problem distinct from the learning-how-to-write-tests problem.

Third, "I’m also going to touch on Perl Tidy and Perl Critic, as ways for us to enforce these coding standards weÂ’re agreeing on." Again, this is a problem distinct from testing.

I would recommend thinking about:

1. Are the developers in my company ready to tackle these three problems simultaneously? If not, how should you arrange your priorities?
2. Can I effectively proselytize for testing, modularized programming and code standards all at once? If not, at which can I be most effective? Are there other developers who would be more effective advocates for one of the projects than I would be?
3. If I am writing tests for already existing code, what happens when my tests begin to suggest that the code ought to be restructured to a greater or less degree? How will the author or current maintainer of that code react? What procedure does your company have set up to handle disagreements as to how code should be structured?

As you can see, these are not Perl questions or testing questions; they are human relations questions. Tests can't change suboptimal code; only humans can. Lest you bite off more than one person can chew, I recommend that you think about these questions a bit, then figure out the best role for you yourself to play.

[NovMonk]

I just wanted to take a minute and thank everyone for the helpful advice. I think we have enough to get us started. I'm sure the learning curve will be steep, but I'll try to keep notes so someday I can share in this space some things geared to folks like me who are inexperienced programmers and first time testers.

Of course, if anyone has more thoughts or ideas, by all means, please share them. One thing I've noticed in my research is that people who have been testing for any length of time, tend to forget the hurdles of those first few days/ weeks/ minutes. I think testing completely changes your habits of thought, and that's in fact what I'm hoping will happen here. But anecdotes from the early days of people's testing careers really are full of insight, at least for me.

Thanks again, brothers and sisters. I'll head back out to the stables now. Got some mucking out to do in my thought processes, I do believe.

Pax,

Novmonk

[McMahon]

Some time ago (March 2005) Kevin Lawrence wrote a great article about how to do incremental development (and testing). It's here

I've used his ideas, and they work.

[NovMonk]

Thanks! Great article. This is exactly the kind of stuff we need.

Pax,

NovMonk