Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re^3: Self Testing Modules

by eyepopslikeamosquito (Bishop)
on Dec 18, 2005 at 21:42 UTC ( #517624=note: print w/replies, xml ) Need Help??


in reply to Re^2: Self Testing Modules
in thread Self Testing Modules

When writing a lot of tests, having one large monolithic block of tests can become difficult to manage. For example, with Test::More, it is more manageable to write many small .t files, rather than one large one.

Replies are listed 'Best First'.
Re^4: Self Testing Modules
by BrowserUk (Pope) on Dec 18, 2005 at 22:01 UTC
    many small .t files

    I cannot tell you how much I disagree with that.

    You're suggesting that it is okay for the module to be a single file, but the tests for that module have to be littered around the file system in lots of tiny files?

    So now to test my module, I need a whole support harness to find, run, accumulate and report on those tests, adding layers to what should be the simplest code possible commensurate with getting the task done, and pushing a damn great wedge between the 'edit' and the 'run' in the development cycle.

    It takes the greatest benefit of 'interpreted' languages, the removal of the 'compile' step, and throws it away by substituting the 'test' step. And for what benefit?

    You now have to sit and wait for it to (re-run) all the tests of all code that you haven't changed, in order to gain a pretty number telling you "XX.X% passed", when what your really want to know is.

    Failure at line nn.

    And preferably, be dropped back into your editor, with that line highlighted. Running any tests after that failure is pointless, because like the warning issued by Perl when you omit a semicolon or brace, only the first one is real and the others are often as not cascade failures from it.

    It makes no sense to me at all.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      Personally I think it makes a lot of sense to split up your tests into seperate files, preferably based on the type of testing that will occur. When your tests start being more code than what is being tested having it all in one file doesnt make a lot of sense. I mean at the very least if your code is 100 lines, and your tests 1000 (a not unreasonably ratio IMO) then if you embed your tests in the same file thats 900 lines of code that will be read and parsed and compiled for no reason every time you use the module.

      And frankly if you are using the test framework to get a pretty number telling you "XX.X% passed" then you aren't using the test framework properly. The issue is to find out what failed not what passed. Having the tests broken down into test files grouped in some sensible way, a failure from a given file can itself be enough information to start investigating. Having the actual test name is even better.

      Anyway, i dont expect youll change your views based on my comments, but hopefully other readers will learn something from this.

      ---
      $world=~s/war/peace/g

        ... if your code is 100 lines, and your tests 1000 (a not unreasonably ratio IMO) ...

        If you are writing 1000 lines of tests to verify 100 lines of code, your code is testing your tests, and not the other way around.

        And frankly if you are using the test framework to get a pretty number telling you "XX.X% passed" then you aren't using the test framework properly.

        It isn't a case of what I'm doing, it's what they do--and the reason I don't use them!

        Anyway, i dont expect youll change your views..,

        Nor do I expect others to eshew de rigueur based upon my rejection of it, but counterpoint is a goood thing wouldn't you agree?


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

      From Perl Testing: A Developer's Notebook by chromatic and Ian Langworth, page 40:

      There's no technical reason to keep all of the tests for a particular program or module in a single file, so create as many test files as you need, organizing them by features, bugs, modules, or any other criteria.

      In my experience, this is sound advice. Dividing a large number of tests into a number of smaller (cohesive) units, is essentially just divide and conquer, the only fundamental technique we have to fight complexity. It also helps ensure that each test runs in isolation, independent of others. To give a specific example, notice that WWW::Mechanize, written by Phalanx leader petdance, contains 3 lib .pm files and 49 .t files.

      Apart from all that, a single large .t file makes developing new tests inconvenient because after adding a new test to the .t file, you must run all the other tests (intolerable if the single .t file contains stress tests taking hours to run). Or are you recommending that we don't use the standard Test::Harness/Test::More framework?

        Or are you recommending that we don't use the standard Test::Harness/Test::More framework?

        I'm not recommending anything, which may be reason enough of itself to ignore what I am saying.

        Which is, that a test framework that forces greater complexity in the unit tests than is required by the code under test has to be suspect. Test code is still code and should be subject to the same rigeuers as any other code. We rightly reject repetitous, c&p code in applications and modules in favour of once and once only. Why tolorate it in test code?

        Using the WWW::Mechanize as an example, as you brought it up rather than because it is a bad one. In fact, I suspect that it is a rather good example of it's type. (*)

        From 1700 non-blank lines spread across those 49 files, a simple de-duping reduces them to 888. That means 812 lines are duplicated! Taking a few simple measures to standardise non-differenciating aspects like whitespace and punctuation, further reduces that tally to ~800, giving 16 lines per file.

        More importantly, over 50% of the lines are duplicates. That's without removing comments or variations in variable names being used for the same purpose, or variations in text constants used to report the same pass/fail criteria etc.

        Over 50% of the code in the test suite are repeatitous, mostly construction code that would be only be required once if the tests were in one file.

        At ~800 line of actual test code, testing 954 non-blank, non-pod, non-comment lines in the 3 modules, the ratio of tests to code seems about right. But 1700 to do the same job, seems kind of wasteful? We wouldn't tolorate that amount of duplication in our application code.

        (*)Please don't take these figures too literally. The were derive using mechanical means without verification of the total accuracy of the processing used. I may have made mistakes. They are just to serve as an indication of the source and direction of my concerns, not a accurate accessment this particular module and its test suite.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://517624]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (8)
As of 2021-06-23 08:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What does the "s" stand for in "perls"? (Whence perls)












    Results (117 votes). Check out past polls.

    Notices?