Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

run tests only changed files

by mandog (Curate)
on Aug 07, 2015 at 13:35 UTC ( #1137849=perlquestion: print w/replies, xml ) Need Help??

mandog has asked for the wisdom of the Perl Monks concerning the following question:

My test suite takes about 6 minutes to run. Most of the time, I'm testing things that haven't changed.

Is  make the tool to run my tests faster?

Does a script to regenerate my makefile already exist?

Translating My::Foo:Thing to 'lib/Foo/Thing.pm' doesn't look very hard. The Test::WWW::Mechanize tests aren't as obvious but eyebballing the output from Devel::NYTProf or Devel::Cover would be a start

Obviously  make clean would be a good idea before pushing to production

Replies are listed 'Best First'.
Re: run tests only changed files
by Your Mother (Bishop) on Aug 07, 2015 at 14:13 UTC

    make doesn't do anything about test speed. The App::Prove tools can help you though. Make sure you have a relatively new version of prove installed and read up on all its options with a focus on these–

    -j, --jobs N Run N test jobs in parallel (try 9.) --state=opts Control prove's persistent state. --rc=rcfile Process options from rcfile --rules Rules for parallel vs sequential proces +sing.
Re: run tests only changed files
by Monk::Thomas (Friar) on Aug 07, 2015 at 13:59 UTC

    My test suite takes about 6 minutes to run. Most of the time, I'm testing things that haven't changed.

    Are you sure nothing has changed? 'Action at a distance' comes to mind. I run tests in order to prove that nothing has changed. Maybe you can optimize the tests and make them run faster? If that is not an option then you could run a specific test file, e.g.:

    perl -I lib/ t/10_lulu.t

    ...and as soon as it yields the expected result run the whole testsuite.

Re: run tests only changed files
by hippo (Canon) on Aug 07, 2015 at 14:01 UTC
    Most of the time, I'm testing things that haven't changed.

    Most of the time you are testing things which you think haven't changed. There's a difference.

    If you really only want to test the bits which you think have changed, then just run those test files. How many files are there in your test suite? Just run the one which tests what you think has changed and only run the full test suite when the individual script passes. Or have the full test suite run on-demand but in the background so you can get on with other things. There are so many ways to approach this without going to the complexity of dependency-driven tests.

Re: run tests only changed files
by CountZero (Bishop) on Aug 07, 2015 at 16:20 UTC
    You don't even have to change things yourself for matters to go wrong. Perhaps some sys-admin has "updated" something somewhere or changed some network configuration and all of a sudden your scripts stop working. You really want to find out before you have to run that business and time critical script.

    CountZero

    A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

    My blog: Imperial Deltronics
Re: run tests only changed files
by (anonymized user) (Curate) on Aug 07, 2015 at 14:33 UTC
    Yes, makefiles can be written to do just that. The secret is to define rules for separately named subtargets.

    Update: re how much to test, testing only changed file impact is fine for unit testing of changes. But when you are regression testing them, sundial's comments below will of course apply. But even if that were true, you still don't need to rebuild targets whose dependencies haven't changed. The exception is of course where you updated the makefile, then you have to clean and rebuild everything!

    One world, one people

Re: run tests only changed files
by sundialsvc4 (Abbot) on Aug 07, 2015 at 20:05 UTC

    Quite honestly, I must insist:   “run it all, always.”   You will be very glad that you did ... and the moment when you will fully appreciate the wisdom of this, will be the first (but, not the only ...) moment when you are “surprised.”

    While constructing an online reservation system for an education company, I religiously adhered to the practice of constructing .t files (and directories) to support each step in the project plan.   The code was changed, then prove was run, then the branch was checked-in and the that ticket was closed.   So far, so good, mmm?   Yeah, pretty routine.   Look out the window while the first fifty tests ran ...

    Until the day inevitably came when one of the earliest tests suddenly failed.   Mystified, I looked at the deltas and discovered the one-byte change that had accidentally just been made ... a change that did not cause the source code to fail-to-compile(!), but that did, in a most-insidious way, break the code.   In that instant, all of us knew why “test-driven development” was so-o-o-o... worth-it, and why this and every other team that I have lead or been on has practiced it ever since.   (Believe me, that bug could have been such a money-sinking b*stard ... had it survived.)

    “Every test, every time.”   Have one part of the team write tests on the other part’s work, with and without consulting the others.   If you plan to make money at this most-crazy game, you’ll be glad you did.   In every software system, there are just so damned many subtle interdependencies, running throughout the code, that you really don’t know and really can’t predict just where all of them are.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1137849]
Approved by toolic
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (8)
As of 2019-06-25 12:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Is there a future for codeless software?



    Results (105 votes). Check out past polls.

    Notices?