Writing a Test Harness

elTriberium has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Writing a Test Harness by trwww (Priest) on Mar 30, 2011 at 23:53 UTC
I know my life would be a lot more difficult without Test::Class	[reply]
Re^2: Writing a Test Harness by elTriberium (Friar) on Mar 31, 2011 at 00:07 UTC
Thanks, that's exactly what I'm looking for, suggestions for other modules.	[reply]
Re: Writing a Test Harness by stvn (Monsignor) on Mar 31, 2011 at 01:33 UTC
I am not sure if it is related but the has been a lot of CPAN activity in the Tapper namespace lately from AMD and it seems to me from the docs to be a large scale test-harness of some kind. You might want to do some code diving and see if it is useful. -stvn	[reply]
Re^2: Writing a Test Harness by Corion (Patriarch) on Mar 31, 2011 at 06:36 UTC
Indeed. It is not really clear from the documentation of Tapper, but I believe this is mainly the work of renormalist who uses it to test and benchmark OSes and search those results in his work for AMD. So, likely, Tapper is of interest to you if you want to run and aggregate various tests on multiple machines and environments.	[reply]
Re: Writing a Test Harness by knbknb (Acolyte) on Mar 31, 2011 at 07:40 UTC
I just learned on stackoverflow.com (sorry, perlmonks.org) that Test::Fatal is preferable to Test::Exception because it is simpler and has fewer dependencies. Read this, written by the creator of T::F, for more: blog post	[reply]
Re^2: Writing a Test Harness by chrestomanci (Priest) on Mar 31, 2011 at 14:17 UTC
That blog post is one person's opinion. rjbs makes a fair point about Sub::Uplevel being a risky thing to have about unless you are sure you really need it. (A bit like having an bottle of deadly poison in your food cupboard), and by extension it is not a good idea to have it used in standard testing code that uses Test::Exception, that otherwise has no need for it. However, he then uses that as an excuse to write a totally new and non standard exception testing module with an incompatible and less expressive API. I think that rjbs would have better spent his time creating a patch for Test::Exception that made Sub::Uplevel optional, so that the danger could be removed without creating yet another perl module that mostly replicates existing functionality.	[reply]
Re: Writing a Test Harness by Anonymous Monk on Mar 31, 2011 at 08:51 UTC
Try to avoid using forked processes in the tests (or to use modules for the test using such processes). If forked processes are used, these must terminate themselves. In Windows forked processes are implemented as pseudo-processes. To use kill(9, $child) on pseudo-processes is unsafe and there is a probability that the process which implements all the pseudo-processes is blocked. The test can hang. The outcome of kill on a pseudo-process, depends on the timing in the operating system, and code that has worked, suddenly can fail, resulting in errors which are difficult to find.	[reply]
Re: Writing a Test Harness by sundialsvc4 (Abbot) on Mar 31, 2011 at 16:38 UTC
Ahem... some folks do tend to write what first comes to mind. Nevermind. I think that it would be helpful if you’d elaborate a little bit on what approach you want to take. In other words, “if I just now stumbled upon ‘the perfect test environment for my present purposes,’ how would I recognize it? What, exactly, would it look like? What, exactly, would it enable me to do that I cannot so easily do today?” Also elaborate on just what this “application” looks like. That word, after all, could mean absolutely anything. When you say, “screens,” how are those constructed? And so on. Please put some serious thought into it. The more thoroughly you are able to answer that hypothetical question, the more focused your search will be – and the more effectively we will be able to help you (and to learn from your experiences). There are, as you have seen, almost a Roman legion of `Test::` modules. All of them are devised for particular purposes, to be useful in particular contexts (and hence, possibly useless in others). You have, indeed, touched upon some of the most useful ones for certain general purposes ... but do they apply to yours? Aye, there’s the rub, and only you can decide it. That “less than diplomatic expression” accurately describes what will happen if you don’t.
Re^2: Writing a Test Harness by elTriberium (Friar) on Mar 31, 2011 at 18:26 UTC
Thanks for your comment, I addressed some of your points in node Re^4: Writing a Test Harness. I understand that I should have given some additional information in the original node, but my reason for not doing that was that I wanted to get some high-level information about Perl Test:: (and related) modules and how people solve some of the common problems.	[reply]
Re^3: Writing a Test Harness by Anonymous Monk on Apr 01, 2011 at 02:57 UTC
Perl testing is not a product, or a package; it is a process. Your comment is simply too vague to be answerable. If your application is not a Perl app then it seems to me that certainly most of the packages that have been listed for you will not be useful to you. Good luck.	[reply]
Re: Writing a Test Harness by educated_foo (Vicar) on Mar 30, 2011 at 23:22 UTC
Your list looks like a cluster-f*** in the making. I suggest trying to keep it simple, with Test::Simple or Test::More printing your test results and `system "ssh user@host perl < script.pl"` to run remote tests.	[reply] [d/l]
Re^2: Writing a Test Harness by elTriberium (Friar) on Mar 31, 2011 at 00:06 UTC
Any explanation why you think it looks "like a cluster-f*** in the making"? I'm fine with other opinions, but some explanation would be nice. I understand that keeping it simple is a good idea, but I don't see why the modules I mentioned above (which are all CPAN modules used in many projects) are particularly complicated. Just to explain some thoughts: I want to use some of the more advanced testing options that are in Test::Most. I don't see how that's worse to using Test::More. There need to be some configuration files, that's why I'm thinking about using YAML, a good, easy to read and write format that translates well into Perl scalars, arrays and hashes. I want to use Moose to actually make the design a bit simpler, by breaking the code down into several classes. Expect (or something similar) will be needed for this project, as it requires to run many commands on different clients and servers.	[reply]
Re^3: Writing a Test Harness by chrestomanci (Priest) on Mar 31, 2011 at 14:43 UTC
I don't know what educated_foo was thinking when he wrote that your plan looks like a cluster-f*** in the making, but I agree, and these are my reasons. I think your are conflating several sorts of testing and monitoring, and trying to coerce the various Test::* modules on CPAN to do things they are not designed for, such as testing your deployment or monitoring. Firstly, the testing modules such as Test::Most are mostly there to test the correctness of your low level code. (Known as API testing) For example suppose in the source code for your project you have an `add_numbers()` subroutine, then somewhere in the corresponding test suite you ought to have: `use Test::Most; is( add_numbers(2, 2), 4, 'Correctly added two numbers' ); throws_ok( sub{add_numbers(2, 'string')}, qr/not a number/, 'Error wit +h non number');` [download] If you write lots of API tests in your test suite, and you use a coverage tool such as Devel::Cover to make sure that all your source code is checked by tests, then you can be reasonably confident that the source code is correct and has no bugs, or at least that any remaining bugs will be at a more conceptual level and are reflected in the design of the tests. (For example if the specification is unclear, then it is likely that you will get both code and tests that agree on the wrong thing.) In general it is a mistake to set-up complex situations such as running commands through ssh sessions on other machines unless that is totally unavoidable. Remember you are testing the correctness of the code, so you want to only test one thing at once, and you want your tests to be simple so that when they fail it is easy to work out what API is broken. Take the example of the `add_numbers()` subroutine. You might expose it though at SOAP API, so you could use SOAP to test it, but if that test failed, you would not know if it was SOAP that was broken in some way, or the API itself, so it is much better to create two simpler tests that test the two separately. Secondly there is system level testing and benchmarking. For application programs (think Microsoft Excel), this is usually done by humans running the complete program and checking that it all works, nothing breaks or looks strange, and the performance is OK. For some types of programs (eg web applications you can automate some of the testing with tools like Catalyst::Test, but it is important to remember that automated testing will only test the things you can think of. Sometimes it takes human randomness to uncover things that the programmer did not think of, or user interface disasters that make perfect sense to a programmer but are impossibly confusing to an end user. You comment about running tests through SSH sessions hints that your are looking to test the deployed application. If that is the case, then you should probably be looking into monitoring tools such as Zabbix or Nagios.	[reply] [d/l] [select]
Re^4: Writing a Test Harness by elTriberium (Friar) on Mar 31, 2011 at 18:08 UTC
Re^3: Writing a Test Harness by educated_foo (Vicar) on Mar 31, 2011 at 02:31 UTC
It seems to me that you're best off with the simplest testing setup possible. However, you seem to be reaching for a bunch of "cool" modules. It makes me wonder what kind of sophisticated testing framework you will use to test your sophisticated testing framework.	[reply]


Welcome to the Monastery
	PerlMonks