Here's a basic list of the discrete tests:
in reply to Re^3: Test::Class and test organization
in thread Test::Class and test organization
I then want to cross-reference those tests against:
- Basic functionality (Does a single level hash or array work?)
- Embedded functionality (Does a HoH, HoA, AoH, AoA, and deeper work?)
- What about wide hashes and arrays (4000+ keys)?
- What about deep hashes and arrays (4000+ levels)? Does every level behave appropriately?
- What happens if we use filters on the keys? The values?
- What about changing some of the internal key values?
- Does changing the hashing function make a difference?
- Does cloning work?
- What about importing and exporting from standard Perl data structures? tied datastructures?
- How about if I turn locking on and off?
- What about creating the db using tie vs. DBM::Deep->new?
- How about concurrent access?
- What about turning autobless on and off? What about locking? autoflush?
It's almost a N-dimensional matrix of possibilities, and I want to be able to cover as many as possible.
Now, 99.9% of these tests will not be run when the user installs. I plan on picking some representative examples that provide 95%+ code coverage in under 30 seconds on a standard machine and using those for installation tests. That suite will also be the tests that I run on a standard basis before committing changes.
However, I need a suite of tests that I can run overnight (if necessary) that will completely and utterly crush this code and show me that every single edge case I can think of has been covered. Then, when a bug is found, I can demonstrate the the bug has been covered in every one of these scenarios. I need to do this so I can have the level of confidence in DBM::Deep that I have in MySQL or Oracle.
A perfect example is the autobless feature. I found a few bugs in autobless in a few situations, so I fixed them. Recently, however, I found a bug with autobless when it came to exporting. Had I been using this comprehensive test suite, I would have found that bug already (and others like it, I suspect).
My criteria for good software:
- Does it work?
- Can someone else come in, make a change, and be reasonably certain no bugs were introduced?