In an attempt to get around the problems with QA described in The purpose of testing, I have been working on something very similar. The chief difference between what you are describing and what I've been working on is that I am seeking to generate automated tests and formal test scripts (in MS-Word) from the same metadata.

I did very much what you are describing, wrote the automation steps first and the tests later. Also, the distinction you make between steps failing and tests failing is one that I've followed.

Probably the most significant thing I found was that the automation steps ended up being reused for all sorts of things - in fact, the test framework is not in use yet, but I've written several support tools (for component & log file monitoring, enabling/disabling components etc) using the automation classes which save me a lot of work from day to day. Since the automation classes also get much more use this way, they've improved considerably in error detection & recovery in ways that they might not if they had only been used for testing.

For that reason I'd recommend not coupling your UI with the automation - instead have packages containing automation functions that return success or failure, and write the user interaction code separately - possibly as part of your test script.

I also had a similar problem of convincing other people this was a good way forward - I got around that by designing the whole thing up front, then initially only implementing those classes I needed for a simple demonstration test case to get buy in.

Much of what you need is built into perl (backticks to run command line utilities, normal filehandling for reading log files, pattern matches for spotting log messages etc) - which is one reason why perl is ideally suited to the task.

If you are going to have a very large volume of tests and associated test data, I'd suggest keeping the test and it's associated test data together for maintainability. I initially did this by storing them both in a database, but later found that XML files were more flexible and easily maintained. If you use the Test::Harness framework for your tests (and I see no reason not to - I didn't really have the option) you could perhaps put the test data for each file in a __DATA__ block, or in a separate file with the same base name, or some other mechanism, as long as there is an easily spotted connection between them.

---

---

• Are you posting in the right place? Check out Where do I post X? to know for sure.
• Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:

  <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>

• Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
• Want more info? How to link or How to display code and escape characters are good places to start.