Automation using perl

[jaypal]

Hello Monks,

I am a quality assurance engineer by profession. I have written a few perl scripts that have reduced my manual testing time considerably. I want to extend it by creating an automation suite that would kick off and validate all my test cases and provide me a summary of which test passed or failed and why.

CPAN lists great amount of testing modules and I had picked up a copy of Perl Testing: A Developer's Notebook to see how I can use these modules to my advantage.

The book was great and the examples were explained well. However, they all showed how to use module to test the perl code.

In my case the application I have to test is a backend application written in java. I will give a small example of what I have to test to make things clear:

I have an application where one of the services is a FTP Service. Core functionality of this service is to get file from remote location to local server. There are 30 different properties for this service (to make it highly configurable) that I have to test. For example:

- To ensure FTP service only picks file with specific naming convention from remote location. The naming convention is specified as part of a property value

- To validate FTP service renames the file with specific prefix on local server. The prefix is picked from a property value.

- A property whose value is either LIFO or FIFO which ensures FTP service picks the most recently created file (LIFO) or the oldest file (FIFO).

- Validate entries of the file in MySQL database

etc ... etc ...

I was planning to write one package script which would contain all my reusable code and then multiple perl scripts for each test (calling it "001_ftp_test.t", "002_ftp_test.t" ...) which would use my package script and finally a master script that would execute either all or specific tests as required. The individual scripts would include testing of all positive and negative scenarios for one particular test case only (to make things easier to maintain in the long run)

My question is how do I use modules like Test::Harness, Test::Builder or Test::More to my advantage. Are these modules only for testing perl code? Should I not use any modules and just write stand-alone perl scripts and then run them as per requirements? Should I create TAP output from my perl scripts in order to use these modules?

Please leave a comment if any part of the question is not clear and I will update the question to fill the gaps. I look forward to your guidance.

Thanks in advance
Jaypal

[Anonymous Monk]

Are these modules only for testing perl code?

No, they are fairly general-purpose. Ignoring the internals, the simplest tests just do a boolean check (e.g. ok() from Test::More) and generate TAP output. You can and probably should use these modules to generate TAP output for you.

use Test::More;
ok connect_to_ftp(), "connect";
is get_something(), "expected string", "get_something";
is system("external command"), 0, "external command";
...
done_testing; # or use "plan tests => 123;" above

Here, connect_to_ftp() can do whatever you want, be it shelling out to some external command, calling a Java API, etc. - as long as it returns a true value on success and a false value on failure you can use it in ok() (even dieing is fine, since Test::More will report that as an error too). Same goes for get_something(), it can read from a socket, file, whatever. If you're going to be doing a lot of calling of external commands, I can highly recommend IPC::Run3 (one of the few modules that handles redirection of all three STDIN, STDOUT, and STDERR streams, and according to the test stats does it well).

Your idea to have a set of *.t files is a good one, this is how just about every module on CPAN with tests does it. Then, you can use the prove command and all the other tools that handle Perl and TAP tests. So you probably don't need a "master script" that executes your tests.

Although this is a design issue, personally I would limit your "test library" to be as small as necessary, since if you end up writing a big test library, you really should write tests for that, too. Instead, make your test cases as small and self-contained as possible (granted, that's not always possible).

(By the way, have you seen Inline::Java, maybe that's useful to you?)

[vinoth.ree]

Hi jaypal,

I have written a Perl framework which uses those modules which you mentioned in your post, my framework will take the test cases as YAML file format. my framework is well suited to test CLI based commands.

YAML file is nothing but has command to execute, output to match. So when my framework runs it takes the command from YAML file and executes and matches with the expected output you configure. so if it matches it passes otherwise fails.

Finally it gives an Excel file consist the passed and failed test cases with status and why it failed like what was expected and what was the actual output. If you are interested in this please get touch with me. I can guide you.

[perlfan]

There was a good talk at YAPC::NA this year that touched upon "testing anything". Steven Lembark - ‎Getting testy with Perl‎.

[sundialsvc4]

Jaypal, my experienced opinion on this matter is that you probably should refactor everything “one more level-of-detail down” in your overall testing hierarchy.   Instead of putting “use my package script” at the outermost level, allow each of the individual test scripts to do it.   (Or, if multiple scripts have common prerequisites, let each of them use a common library which, in turn, specifies the individual uses.)

When any particular (outer-level) test script finally runs, then from the Perl interpreter’s point-of-view it will “top down, incorporate everything.”     However, generally speaking, you would prefer to be able to also run any individual script, well, “individually.”   Therefore, it’s generally a good idea to put all of the prerequisites for each individual test-case at the bottom of the tree, so that each case, if run individually, is guaranteed to produce the same results that it would when it is run within the larger framework.

Therefore, build each script, necessarily “from the bottom up,” in such a way that it will perform identically when run “from the top down.”   If a group of tests share common prerequisites, let all of them use a common ancestor which, in turn, uses all the rest.

[Anonymous Monk]

Without code or a more detailed description, it's practically impossible to tell where and what the "top", "bottom", and "outside" of this "tree" you are describing is... perhaps a diagram would help?