Philosophical question about testing

[rastoboy]

So I'm writing my first serious test script for a module I've written, and I'm finding that I'm writing the same code in the test script as I did in the module.

Does this make me a bad person?

I realize that part of the point of a test is just to make sure the module is complete and in good shape. But it seems wrong to use the same logic. For example, in my module I stat a file to see if the permissions are acceptable, and if not, chmod it. So in my test script I create a file with bad permissions, run my module function to fix it, and then use basically the same code as I did in my module to see if the new permissions are acceptable.

I do hope this makes sense. It just seems to me that I'm not testing my logic very well in my test script if I'm using the same logic as I used in the module--of *course* it works great!

Any input would be greatly appreciated!

[Old_Gray_Bear]

You said: "Does this make me a bad person?"

Not necessarily bad, just cautious -- it doesn't feel right -- and pragmatic, you went and asked for a second opinion.

When I am writing tests I try as much as possible to get into the 'assertion' mind-set -- is the value returned from my method equal to 'X'? Simple, yes/no tests.

If my method has side-effects (I call it with Y, I get X back, and by the way, the file is now readable....), I'd think about alternative ways of testing my side-effect. Say, create a file without read-permission, call my routine that sorts out the permissions bits to what I want. Then open the file for reading and check the return value from open() ('true' on success, and 'false' on failure). This way you could write a binary test for a successful open. Any function that you can use the  or die ... idiom on is susceptible to this approach.

If you haven't run across Ian Langworth and cromatic's book Perl Testing: A Developer's Notebook, you should. They have a number of recipes for testing most everything under the Sun using Perl and the Test Anything Protocol.

[toolic]

Does this make me a bad person?

No. The fact that you are writing tests makes you a good person.

It just seems to me that I'm not testing my logic very well in my test script if I'm using the same logic as I used in the module--of *course* it works great!

You have good instincts. You should strive for an orthogonal approach, rather than using a copy of your module code to check your module code. Perhaps one of the CPAN Test modules would be appropriate. A quick look at Test::File looks relevant.

I realize that part of the point of a test is just to make sure the module is complete and in good shape.

The point of a test suite is to make sure the module does what you want it to do, and more specifically, what you said it would do in your public documentation (POD). At a minimum, it should perform basic checks of all the features in your doc. More thoroughly, it should check edge cases of your well-specified input ranges. Then it should check how your module reacts to illegal inputs.

[doom]

You have good instincts. You should strive for an orthogonal approach,

This is true, of course, you should try to think of a way to independently verify what the code does, but I would suggest there isn't any reason to go beserk about it.

Consider the fact that the algorithm in both code and test may be the same to start with, but they may diverge later. You might go in and refactor the main code, for efficiency, let's say, and then the older style of code out in the tests will be an excellent check on your work.

In general, tests should be relatively easy to write, or else you (or worse, your boss) will burn-out on it, and start skipping them altogether. A lot of things that would be Bad And Wrong in your real code are okay in your tests... e.g. cut-and-paste is a much more valid approach.

[eyepopslikeamosquito]

Did you write the test script before or after writing the module? Writing tests early accrues many benefits.

I think it's appropriate that you're using the same (perl) file permission techniques in the module and the script; that makes the test script more portable and easier to maintain than if you checked file permissions using Linux or Windows specific techniques, for example. After all, you're not testing perl itself, you're testing your module: you want to demonstrate, via your test script, that your module has a nice, clean, portable, easy to understand interface, and is easy to test in isolation. Testing file permissions in a non-perl way, while a good idea for testing perl itself, introduces unnecessary complexity and an unwelcome dependency for your module's test suite.

[aquarium]

just from what i've seen...the test harness would use chmod (for example) which works equally well on linux and unix, but windows is a different beast of course for most systems that don't have cygwin installed...our systems did. anyway, the point of using system utilities instead of perl built-ins is to more closely approximate system conditions in a production environment. whilst i wouldn't preclude totally using some perl in a test harness, i think you need some degree of separation from test harness and app/module being tested. otherwise you're writing just a slight variation of the original script, instead of a test harness, i think.
a good example of test harness methodology and code is the various test/install scripts in unix/linux software, or even the unix/linux oracle installer scripts.

[sundialsvc4]

I am working on a fairly complicated set of parsing modules, and as I have written each one of them I have written a set of test-scripts.   I know that the modules passed all of those tests when written, but it is of course devilishly easy to “break something else” in this curious china-shop that we call our daily workplace, especially when you are parsing things and writing a lot of regular expressions.   It is therefore very useful to have a known set of things that “this code has been known to do without error,” and the means to test automatically that it still does.   (e.g. prove -r t/*)   You’d be surprised just how often it doesn’t work anymore ... and if you were not consciously looking, you wouldn’t know.   Yet.   Heh.

This has allowed me to avoid a lot of red herrings.   As you move forward through the writing process, and you (inevitably) discover another bug, it is natural to assume that it must be in the new stuff; that what was working before is still working.   But, more often than not, that is not the way it works out ... and the time-savings can be huge in the long run.

When you think you’ve got it finished, that’s when you start writing more tests ... what I call the “kick the tires and look under the hood” tests.   But that is not the time when you should begin writing tests.

[tospo]

You could also address your concerns by refactoring your module a bit. Your module could have a subroutine/method to check that a file fulfils the criteria needed to be used by the module, whatever they are. Then there could be a separate sub/method that attempts to make a file compliant with your requirements and return "true" if it suceeds and "false" if it doesn't.
In your test script you would now create a good file and a bad file and the test would first assert that they are picked up accordingly by the checking-method.
Then you would pass the bad file through the secnod method and assert that it returns true. You should also create a file that can't be fixed and assert that your second method returns false on that one, meaning that the error can be picked up and handled.
Now you don't duplicate any code anymore, you just assert that your methods return the right thing.
You would then have something like:

ok( not file_is_usable( $bad_file ), "a bad file is identified correct
+ly);
ok ( make_file_usable ( $bad_file ), "can correct a bad file")

In addition, you have also made your methods more focused on one thing and reduced side-effects.

[Narveson]

Yes, your question makes good sense. It sounds to me as though your test is still operating at too high a level. With experience, you will write smaller tests, each of which confirms some concrete outcome.

You say your test suite uses basically the same code as in your module to see if the new permissions are acceptable. So separate the checking and the fixing.

Create some files and set their permissions explicitly and test whether your permission checking code, at run time, evaluates the acceptability of each set of permissions in the same way that you, at test writing time, believe it should. Nothing circular about that.

[aquarium]

for coverage testing you typically provide not just the input that tests the limits, but you exceed the limits also. you may possibly be assuming that a successful chmod gives you what you want...what if someone deletes the file just after you chmod it? what happens if the module is supplied with garbage as input?..or supplied with null device as a filename instead?
from what i've seen so far in test scripts, perl merely provides the conditional and loop constructs for test cases (the harness), but most of the tests and heavy lifting is with suitable system commands.

[Monkomatic]

One of the few quirks with perl is that:

The 20 line code you wrote 4 weeks ago.

Becomes a 15 line code 2 weeks later.

Then 1 week later you discover how to do it in 2 lines.

And then someone (usually on this very forum) points out that it is possible in 1.

This is why modules and subs arent talked about much

So I wouldn't really stress the Philosophical side of modules. Do it to help understand how to share your code with the community when you have code you wish to share. (CPAN)

[dsheroh]

While your given "quirk" is true (TIMTOWTDI and all), I draw exactly the opposite conclusion from it: When there are many ways to do something, the philosophical side helps to decide which is the appropriate way in any given situation.

More relevantly to the original question, being likely to change the implementation of a method over the lifetime of the code means that it's less of an issue if your testing code mirrors the (original) implementation, since it will still assure you that the method produces the same results after its implementation is changed. (But, that said, I agree with earlier responses that this particular case sounds like one in which breaking the code down further into finer-grained methods and testing those smaller pieces independently is the way to go.)

[doom]

This is why modules and subs arent talked about much

Modules and subs are talked about all the time by perl programmers who have half a clue about what they're doing.

If you push a block of code off into a sub inside a module, among other things you'll discover that it's much easier to write tests for it, and once you've got tests for it, it's much easier to re-write it as you describe (though reducing line count is hardly ever the first priority).

[pemungkah]

As said previously, you're a great person for wanting to write tests.

If you find you're rewriting your code to test your code, you haven't stepped far enough outside the mindset of your program yet. Your tests want to make sure all the parts work, so you need to figure out how to make each part work in isolation.

Let's say you have a main program that calls read_input(), calculate(), and write_output(). What you need to test is does each function do what it is supposed to? and if the functions all do what they are supposed to, does the main program work?

For instance, let's take write_output(). It should take some existing data and write that out to a file. To make this easy to test, you'll need two things: a way to provide data to the function, and a set of example output files to check the function's output against.

Let's assume that you pass a data structure and a file handle in to write_output() (this arrangement makes it particularly easy to test, since we have control over where the input comes from and where the output goes - writing code to be easy to test is never a bad idea!).

Now start enumerating the possibilities - what output should I get if I pass undef? a null string? an empty array/hash? a data structure containing one good entry? (use Data::Dumper to get the data structures to pass in, and use File::Temp to capture your output and Test::Differences to validate what you got vs. what you want).

You can test the input function and the calculate function in much the same way: given these inputs, what outputs do I get? Test::Deep and is_deeply from Test::More are both useful (Test::Deep is more detailed) to compare two data structures.

That question - "given this input, do I get the expected output?" is the key question to ask when writing a test. It sometimes takes a certain amount of thought to figure out how to set up the inputs and outputs - for example, when testing database functions, it's sometimes necessary to load dummy data into your database and test with that (say you want to check your error handling of bad or missing data - your code may not allow you to create the bad data, but it should be able to handle things if the data is bad for some reason.)

To test your main program, you need to eliminate complications from the functions and just see if the main program does what it should. If the program's been written as a standard script, (i.e., code flows from top to bottom, and all functions are defined in place), you'll only be able to feed it inputs and look at its output to see if it works - it's technically possible to "mock out" the functions (that is, replace them with dummy versions that do a straight mapping from "I got this" to "I return that", eliminating the possibility of logic problems in the functions as contributions to problems - and since you've got inputs and outputs already from the function tests, this should be lots easier to do!), but if you've isolated the subroutines in a separate package both testing and mocking them becomes way easier.

This is also one of the reasons it's simpler to write tests first, because you don't find yourself in the position that you have code that is hard to test, because you've designed in ease of testing from the start.

[tospo]

Very nice write-up by pemungkah above.

Just wanted to add one more thing: when you challange a test with a lot of different inputs I recommend defining the input and expected output in a data structure, such as an array of hashes like this:

my @tests = (
  {
    desc => "a file with wrong permissions can be repaired",
    test_file => {
      { permission => "111",
        contents => "some string"
      },
     is_valid => 0,
     is_repairable => 1,
  },
  { desc => "an invalid file that can not be repaired throws an error"
+ 
    # ADD DATA HERE

  }
);

Now it becomes much easier to test all those different cases in a loop that creates each of those files, gives it the permissions and maybe writes some content into it and then runs the tests for file validity and/or repairability (is that even a word?) on each one in turn without having to copy and paste the test code for each case.
You would feed the is_valid and is_repairable flags into the actual tests to decide which is the correct outcome and the desc would be the description printed by the tests. Adding another case is then as easy as adding another block of definitions to the above datastructure.

[rastoboy]

Wow, LOTS of great feedback, thank you very much quite helpful!