Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

When Test Suites Attack

by Ovid (Cardinal)
on Oct 28, 2005 at 20:23 UTC ( #503758=perlmeditation: print w/ replies, xml ) Need Help??

One of the less appreciated benefits of test suites is how they speed up development. It seems counter-intuitive as you're writing twice the amount of code, but the experience of many programmers has been that test suites lead to better interfaces, less time spent tracking down bugs and, due to easier refactoring, a smaller code base that's easier to work with. I personally find that I'm getting much more work done now that I write solid test suites.

Until now.

The code is designed well enough that adding new features is quick and easy. Unfortunately, whenever I need to change my code I fire up a Web server and view the results in the browser and then write the tests after I've written the code (this is closely related to When test-driven development just won't do). This is because XML and XHTML are just text. I need to see the output. I've been finding more and more that small changes in my code are making huge changes in the output and trying to continuously update the tests to exactly match the XML, XSLT and XHTML using Test::XML and XML::XPath has led to a serious productivity drop.

I'm considering just using Test::WWW::Mechanize to do integration testing through a Web server I run in the tests. This will be much faster and allow me to get my development speed back up. However, I'd be skipping the unit testing of the output. I'll catch the bugs but it will likely take me longer to track them down.

Cheers,
Ovid

New address of my CGI Course.

Comment on When Test Suites Attack
Re: When Test Suites Attack
by pg (Canon) on Oct 28, 2005 at 20:39 UTC
    "One of the less appreciated benefits of test suites is how they speed up development. It seems counter-intuitive as you're writing twice the amount of code, but the experience of many programmers has been that test suites lead to better interfaces, ..."

    Just to add one important benefit: quite often, when a programmer fix one bug, they introduce some other bugs that break existing functionalities. The test suites made it possible to quickly retest all existing functinalities, and quickly realize any of this type of bugs, thus lead to quick fix.

Re: When Test Suites Attack
by jk2addict (Chaplain) on Oct 28, 2005 at 20:57 UTC

    I feel your pain. The test suite for Handel has xml/tt output tests for its AxKit and Template Toolkit plugins. I've got oodles of template pages using the components whos output I compare to static .out files that contain the expected output. Right now, that's about 2000+ tests, with about 200 of those being the actual page output tests using all of the possible tag variations.

    Everytime I write a new plugin, or a new tag in the plugin, I waste tons of time just writing the tests for them. So far, I've been good about writing the tests before I write the code, but it takes forever and I rarely get the tests right the first time.

    I'm curious to see what comes out of your question. I'm in the same boat.

Re: When Test Suites Attack
by friedo (Prior) on Oct 28, 2005 at 21:26 UTC
    I came up with a strategy for this type of thing working on a project that uses CGI::Application and Template. First, unlike the usual CGI::App style, my runmodes return only raw data structures and I save the template rendering for the cgiapp_postrun phase. When running tests, I can override the cgiapp_postun method and set $ENV{CGI_APP_RETURN_ONLY} = 1. Then I can set up a fake CGI environment, load and run my module directly (without going through a web server) and use stuff like is_deeply to make sure the data structure looks right.

    I also have a separate set of tests for making sure the templates render properly.

    Also, since my app has a rather large inherritence tree and lots of small subclasses that only override a method or two, I use Test::Class to do xUnit style inherrited tests. That way I can set up a new subclass that might only have a dozen lines of code, and set up its test class to inherrit the six hundred tests from the parent class.

Re: When Test Suites Attack
by Anonymous Monk on Oct 28, 2005 at 22:09 UTC
    I've been finding more and more that small changes in my code are making huge changes in the output and trying to continuously update the tests to exactly match...
    I don't know your problem domain, but would it be possible to test for other (higher level) invariants besides an exact textual match? For instance, maybe you could just check that a transformation preserves syntatic correctness, or that all tags are balanced, or properly nested, or conserved, etc. Let's take a different example. If you are creating a ray tracer, you could have a unit test that renders a red sphere on a black background and then compares that bitmap to a correctly rendered red sphere bitmap, checking that each pixel is correct. Or, you could instead write tests like...
    • Render a red sphere. Are any pixels in the bitmap red?
    • Render a red sphere with a large radius. Are there more red pixels than with a small radius?
    • Render a red sphere that is far away. Does it produce fewer red pixels than one that is close to the camera?
    • etc.
    Does that make any sense?

      I'm taking a roughly similar strategy now. Previously I had many of these:

      is_xml $result, $expected, '... and we should receive the correct XML';

      By generating $expected with a different algorithm (to ensure that I'm not just reaching into buggy code), I had to constantly maintain two different sets of code which did the same thing. It was very painful. Now I'm doing this:

      is_well_formed_xml $result, '... and we should receive well-formed XML';

      And to verify that it's correct, I'm adding more high-level integration testing. It's an annoying trade-off, but like what you're suggesting, it's a reasonable one.

      Cheers,
      Ovid

      New address of my CGI Course.

Re: When Test Suites Attack
by xdg (Monsignor) on Oct 28, 2005 at 22:44 UTC

    It sounds like you may be committing to writing tests while you are still playtesting your solution and specifications. If you're hacking on the output to figure out what you want, and only know if you've found it through visual inspection, then maybe you need an easy way of capturing/saving the output as the specification because that really is the spec. You can bootstrap it by writing your output directly and then tweak and enhance.

    The other thing that occurs to me is that it your tests may be too tightly coupled and you may want to break them down to greater granularity for unit testing. (I.e. just like testing subroutines rather than a whole program.) If you have a template and pass a bunch of stuff to it, write the test for how you want the data structure to be and test that. Then test the XSLT transformations separately. Do you really need to test it end to end on a piece of data?

    Put differently, how much does the success of one test depend on the success of the other? Are you re-writing so much of your tests with each change because the tests aren't sufficiently independent? (Some additional discussion on independence, redundancy, and coupling is in Functional and Unit Test Redundancy)

    -xdg

    Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

Re: When Test Suites Attack
by BrowserUk (Pope) on Oct 28, 2005 at 23:39 UTC

    I would start with a well-formed document of the same type from another source that contains one or two elements of each type that my api can produce. Trim this document manually to remove excessive duplication--and check it is still well-formed. My test application would be to reproduce this document.

    My test would not be a simple textual compare, as the layout of two functionally identical XML docs can vary enormously without there being an error. So, I would use one of the XML parsing modules that produces a dumpable hierarchal object and the test would be comparing the dumped (with sorting where applicable) parse-trees.

    I'd use diff (the program rather than module) to compare the dumped trees. I'd combine two passes with a shell script &&. The first would use the -q switch to just test same/different. And if it fails, it would invoke the second pass that would use the -y switch piped to more* to present the differences on screen, so that I could inspect them and decide what corrections or addition I would make next.

    *I'd probably use windiff for the second pass, its visualisation is easier to digest.

    You can set the test up before you start developing the API and get direct feedback as you go along as to what you have done so far, what corrections you need to make, and what needs to be done next.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: When Test Suites Attack (text; diff)
by tye (Cardinal) on Oct 29, 2005 at 01:17 UTC

    The more I work with Perl test suites, the more I dislike the is_same( $got, $want ) methodology.

    I've toyed with writing Perl tools so my modules could instead use the methodology of t/feature.t (a Perl script), t/feature.ok (the expected output), and "perl t/feature.t | diff t/feature.ok -".

    Then often "fixing" the test suite is as simple as "perl t/feature.t > t/feature.ok" (once you've verified the that the changes are correct).

    Even better is to not use plain 'diff' but something that knows how to ignore or transform variant parts of the output (something that I'll probably write up in more detail at some later date; a combination of simple 'quoting', simplistic templates, and simple reverse templating).

    Update: BTW, this "easy to 'approve' new UT output" feature isn't the only reason I prefer this style of UT validation. It also means that when a test fails, someone can simply send you the output from "t/feature.t" and you've probably got all of the information you'd be trying to find in the debugger (if you could even get to a debugger in an environment where the problem is reproduced) or be trying to figure out what "debugging" prints to add in order to figure it out. By using 'diff' to validate the test, you've already figured out what is important to display.

    It also encourages you to make all of your inner workings have "dump to text" modes, which is often very handy in other phases of maintenance, or even when adding new features. Sure, sometimes Data::Dumper or similar is enough, but custom dumping usually cuts more to the heart of the situation and so is valuable, usually in addition to using a general-purpose dumper.

    - tye        

      This sounds very interesting. I'd love to see more concrete examples of it.

      Cheers,
      Ovid

      New address of my CGI Course.

        I used something simular to described above in HTTP::WebTest's tests. I had to test whenether text output matches given samples. And it too was PITA to update tests each time I change something. The solution was writting these text samples in files and adding special test mode when the text samples are updated from test results. Take a look on my module tests and particulary on HTTP::WebTest::SelfTest (all on CPAN). Here is the relevent bit from HTTP::WebTest::SelfTest. The subroutine compare_output acts more or less like Test::More's is with a difference that it only works for text and the expected result is stored in file.
        sub compare_output { my %param = @_; my $check_file = $param{check_file}; my $output2 = ${$param{output_ref}}; my $output1 = read_file($check_file, 1); _print_diff($output1, $output2); _ok(($output1 eq $output2) or defined $ENV{TEST_FIX}); if(defined $ENV{TEST_FIX} and $output1 ne $output2) { # special mode for writting test report output files write_file($check_file, $output2); } } # ok compatible with Test and Test::Builder sub _ok { # if Test is already loaded use its ok if(Test->can('ok')) { @_ = $_[0]; goto \&Test::ok; } else { require Test::Builder; local $Test::Builder::Level = $Test::Builder::Level + 1; Test::Builder->new->ok(@_); } }
        So my workflow was: change something, run tests, see that it fails as I expect (by inspecting diffs), run tests again in self-update mode.

        --
        Ilya Martynov, ilya@iponweb.net
        CTO IPonWEB (UK) Ltd
        Quality Perl Programming and Unix Support UK managed @ offshore prices - http://www.iponweb.net
        Personal website - http://martynov.org

      Sure, sometimes Data::Dumper or similar is enough...

      It's for things like this that a just-a-dumper dumper is valuable. The last thing you want is for the dumper to arrive at arbitrary decisions about whether the substructure under key A is a duplicate of that under key D on one pass and vice versa on another, because that's the keys came out of the hash in different orders.

      You'd probably have to apply a sort option to get comparible output from a hash anyway, but for this kind of thing you want to see what's there in detail, not an abbreviated reminder that this bit at the bottom is the same as that bit way back up near the top. Especially in heavily self-referencial structures.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
      I've toyed with writing Perl tools so my modules could instead use the methodology of t/feature.t (a Perl script), t/feature.ok (the expected output), and "perl t/feature.t | diff t/feature.ok -".

      This sounds very similar to the kind of test framework you can build up with Test::Base.

Re: When Test Suites Attack
by BerntB (Deacon) on Oct 30, 2005 at 00:44 UTC
    I've toyed a little bit with the idea of a grep(/addition to the RE engine) for XML, which can take flags like this word should be part of an attribute name, return the element name.

    Some kind of diff would be a (hrm..) simple extension of the idea.

    I didn't think it would be that interesting, but here it would find a use. (-: I have a too big hobby module already, anyway. :-)

    Update: What I meant was, a grep thing which understood XML syntax and e.g. allow one grep pattern over the value of an attribute and another grep pattern over the CDATA for that same element.

    I don't really know what would be useful (I don't really need it for what I am thinking of doing) and I doesn't have time now anyway.

Re: When Test Suites Attack
by adrianh (Chancellor) on Oct 30, 2005 at 12:51 UTC
    This is because XML and XHTML are just text. I need to see the output. I've been finding more and more that small changes in my code are making huge changes in the output and trying to continuously update the tests to exactly match the XML, XSLT and XHTML using Test::XML and XML::XPath has led to a serious productivity drop.

    I don't know if this maps to your problem domain - but I was having similar problems with some munging of CSVs that I was doing recently. The flash of the blindingly obvious that I finally got was that I needed to test on a smaller scale.

    So rather that doing the moral equivalent of:

    eq_or_diff( $input, munge( $input ), 'remap actions, dates and ids');

    I moved to:

    is( munged_date($start_date , $date) ); # ... more date tests is( munged_action($start_action, $action )); # ... more action tests is( munged_id($start_id, $id )); # ... more id tests

    with the eq_or_diff relegated off to the acceptance tests.

    This enabled me to progress much faster since I could test the individual tweaks of the data munging bit in isolation.

    I'm considering just using Test::WWW::Mechanize to do integration testing through a Web server I run in the tests. This will be much faster and allow me to get my development speed back up. However, I'd be skipping the unit testing of the output. I'll catch the bugs but it will likely take me longer to track them down.

    Surely you can get your XML out without having to run a W3 server as well (he asks curiously :-) ?

Re: When Test Suites Attack
by dragonchild (Archbishop) on Oct 30, 2005 at 14:00 UTC
    It sounds to me as if you're testing too large a chunk of functionality. You're describing integration tests, not unit tests. This is too big to be a "unit".

    I'd put in a mock XML generator that captures the data structure you're passing through to the thing that make XML/XHTML. Verify that the data structure you're passing to the XML/XHTML generator is correct. Then, I'd test your XML/XHTML generator completely separately. Those are the "units" involved here.


    My criteria for good software:
    1. Does it work?
    2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

      Would you agree that the integration tests are required?


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        Yes, I do. But, the developer(s) involved in the code being integrated shouldn't be doing integration tests. Those should be done by someone else, preferably a dedicated tester (though an uninvolved developer would do). And, integration tests are no substitute for unit-tests, system tests, and user-acceptance tests. Each tests a different view and a different slice of the product.

        My criteria for good software:
        1. Does it work?
        2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
Re: When Test Suites Attack
by exussum0 (Vicar) on Nov 02, 2005 at 16:03 UTC
    Someone spoke about it earlier, but I'll chime in knowing little about what anyone else has said beyond that.

    Hopefully, you've separated out the logic of manipulating some data and performing an action upon it, from the layer. Your unit tests should be as simple as possible, and not dealing w/ your web layer - your presentation layer. It makes your tests simpler w/o debugging the html, the presentation layer AND the business layer.

    It sounds like what you are talking about is interface testing, or end to end testing. Unit testing programmatically will be a bitch. What's worse is, you won't be testing the layout, the JS involved, form elements rendering - text area instead text field, radio button instead of checkboxes. There are tools, though I don't know them off the top of my head, to make IE and Firefox emulate a user browsing around. Something like, http://www.worksoft.com/ContentDisplay/G11/G11L60.asp, though I have no clue of the quality of that particular piece. Just an example of what you should look for.

    ----
    Give me strength for today.. I will not talk it away..
    Just for a moment.. It will burn through the clouds.. and shine down on me.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://503758]
Approved by Corion
Front-paged by TStanley
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (6)
As of 2014-11-23 06:17 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (128 votes), past polls