Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Re^4: Self Testing Modules

by BrowserUk (Pope)
on Dec 18, 2005 at 22:01 UTC ( #517630=note: print w/replies, xml ) Need Help??


in reply to Re^3: Self Testing Modules
in thread Self Testing Modules

many small .t files

I cannot tell you how much I disagree with that.

You're suggesting that it is okay for the module to be a single file, but the tests for that module have to be littered around the file system in lots of tiny files?

So now to test my module, I need a whole support harness to find, run, accumulate and report on those tests, adding layers to what should be the simplest code possible commensurate with getting the task done, and pushing a damn great wedge between the 'edit' and the 'run' in the development cycle.

It takes the greatest benefit of 'interpreted' languages, the removal of the 'compile' step, and throws it away by substituting the 'test' step. And for what benefit?

You now have to sit and wait for it to (re-run) all the tests of all code that you haven't changed, in order to gain a pretty number telling you "XX.X% passed", when what your really want to know is.

Failure at line nn.

And preferably, be dropped back into your editor, with that line highlighted. Running any tests after that failure is pointless, because like the warning issued by Perl when you omit a semicolon or brace, only the first one is real and the others are often as not cascade failures from it.

It makes no sense to me at all.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^5: Self Testing Modules
by demerphq (Chancellor) on Dec 18, 2005 at 23:34 UTC

    Personally I think it makes a lot of sense to split up your tests into seperate files, preferably based on the type of testing that will occur. When your tests start being more code than what is being tested having it all in one file doesnt make a lot of sense. I mean at the very least if your code is 100 lines, and your tests 1000 (a not unreasonably ratio IMO) then if you embed your tests in the same file thats 900 lines of code that will be read and parsed and compiled for no reason every time you use the module.

    And frankly if you are using the test framework to get a pretty number telling you "XX.X% passed" then you aren't using the test framework properly. The issue is to find out what failed not what passed. Having the tests broken down into test files grouped in some sensible way, a failure from a given file can itself be enough information to start investigating. Having the actual test name is even better.

    Anyway, i dont expect youll change your views based on my comments, but hopefully other readers will learn something from this.

    ---
    $world=~s/war/peace/g

      ... if your code is 100 lines, and your tests 1000 (a not unreasonably ratio IMO) ...

      If you are writing 1000 lines of tests to verify 100 lines of code, your code is testing your tests, and not the other way around.

      And frankly if you are using the test framework to get a pretty number telling you "XX.X% passed" then you aren't using the test framework properly.

      It isn't a case of what I'm doing, it's what they do--and the reason I don't use them!

      Anyway, i dont expect youll change your views..,

      Nor do I expect others to eshew de rigueur based upon my rejection of it, but counterpoint is a goood thing wouldn't you agree?


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

        If you are writing 1000 lines of tests to verify 100 lines of code, your code is testing your tests, and not the other way around.

        Heh, well, ok, maybe thats a bit high. :-) OTOH, A hundred lines of code can involve an awful lot of behaviour that needs testing, especially in perl, and IME test code tends to be quite chunky anyway. Maybe stating a hard and fast ratio is not so easy, but i believe the point is still sound.

        ---
        $world=~s/war/peace/g

        It isn't a case of what I'm doing, it's what they do--and the reason I don't use them!

        XX.X% passed is not the only information you get out of a T::B based test suite. It's certainly not the information I'm focused on when I'm developing.

        All I'm looking for is whether the last test I wrote passed or failed. Personally I find that the existing tools give me this info pretty easily. Especially now that Test::Builder displays the test description so that it can be read without dropping into verbose mode.

        The only place I find this information more tricky to find, sad to say, is with my own Test::Class - and I'm working on fixing that :-)

        This isn't to say the pretty numbers and statistical summaries are completely useless. While it's not of much relevance during development, it can be a useful diagnostic tool at install/distribution time.

Re^5: Self Testing Modules
by eyepopslikeamosquito (Bishop) on Dec 19, 2005 at 01:15 UTC

    From Perl Testing: A Developer's Notebook by chromatic and Ian Langworth, page 40:

    There's no technical reason to keep all of the tests for a particular program or module in a single file, so create as many test files as you need, organizing them by features, bugs, modules, or any other criteria.

    In my experience, this is sound advice. Dividing a large number of tests into a number of smaller (cohesive) units, is essentially just divide and conquer, the only fundamental technique we have to fight complexity. It also helps ensure that each test runs in isolation, independent of others. To give a specific example, notice that WWW::Mechanize, written by Phalanx leader petdance, contains 3 lib .pm files and 49 .t files.

    Apart from all that, a single large .t file makes developing new tests inconvenient because after adding a new test to the .t file, you must run all the other tests (intolerable if the single .t file contains stress tests taking hours to run). Or are you recommending that we don't use the standard Test::Harness/Test::More framework?

      Or are you recommending that we don't use the standard Test::Harness/Test::More framework?

      I'm not recommending anything, which may be reason enough of itself to ignore what I am saying.

      Which is, that a test framework that forces greater complexity in the unit tests than is required by the code under test has to be suspect. Test code is still code and should be subject to the same rigeuers as any other code. We rightly reject repetitous, c&p code in applications and modules in favour of once and once only. Why tolorate it in test code?

      Using the WWW::Mechanize as an example, as you brought it up rather than because it is a bad one. In fact, I suspect that it is a rather good example of it's type. (*)

      From 1700 non-blank lines spread across those 49 files, a simple de-duping reduces them to 888. That means 812 lines are duplicated! Taking a few simple measures to standardise non-differenciating aspects like whitespace and punctuation, further reduces that tally to ~800, giving 16 lines per file.

      More importantly, over 50% of the lines are duplicates. That's without removing comments or variations in variable names being used for the same purpose, or variations in text constants used to report the same pass/fail criteria etc.

      Over 50% of the code in the test suite are repeatitous, mostly construction code that would be only be required once if the tests were in one file.

      At ~800 line of actual test code, testing 954 non-blank, non-pod, non-comment lines in the 3 modules, the ratio of tests to code seems about right. But 1700 to do the same job, seems kind of wasteful? We wouldn't tolorate that amount of duplication in our application code.

      (*)Please don't take these figures too literally. The were derive using mechanical means without verification of the total accuracy of the processing used. I may have made mistakes. They are just to serve as an indication of the source and direction of my concerns, not a accurate accessment this particular module and its test suite.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        We rightly reject repetitous, c&p code in applications and modules in favour of once and once only. Why tolorate it in test code?

        I'm in favor of reducing duplication in test code. It's sort of why Schwern and I built Test::Builder.

        Remember though, copy and paste code is only a problem if and when it becomes a maintenance burden. If it is, someone should refactor it -- and perhaps create a new test library either for the project or, if possible, for the CPAN in general.

        If it's not getting in the way though, what's the problem?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://517630]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (5)
As of 2021-06-21 00:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What does the "s" stand for in "perls"? (Whence perls)












    Results (97 votes). Check out past polls.

    Notices?