|No such thing as a small change|
Creating a co-operative framework for testingby gmax (Abbot)
|on Oct 23, 2006 at 10:28 UTC||Need Help??|
I recently applied for a post of QA developer at a open source company and I started working there, with great delight, about one month ago. One of the reasons for my hiring was my active participation in the community, with blogging, writing articles, submitting bug reports, answering to forum and NG questions. My background as a tester was considered, of course, but the stress was on my community links.
Inviting an open source community to Quality Assurance tasks
Thus, one of my tasks is now to organize a framework for cooperation between the Quality Assurance department and the community.
It is not an easy task to tackle. I can see how much an active community can contribute (bug reports, test cases, code patches, usability feedback), but I can also see the clash between a QA department populated by professional testers and the erratic behavior of a large community. Somehow I have to find a way of reconciling these two worlds, and make them work harmoniously toward a common goal.
My plan (yes, in spite of the task being so though, I do have a plan) is to motivate potential contributors with durable benefits, to get the best possible outcome from this relationship.
During the past twelve months, the company has promoted such cooperation with a contest. Win an iPod if you submit many bugs, and/or write articles or blog posts about our products. The results were quite good in the beginning, but they faded away soon after a few weeks. Some of the prizes are yet to be awarded. Why did this policy fail to provide the expected results? IMO, because it was targeting the wrong quality of a potential contributor: it was appealing at the competitiveness, instead of tickling pride and desire of recognition, which work much better in an open source environment. But especially, it did not address the main aspect of open source success, i.e. mutual benefit.
Why does someone contribute to an open source project? There are many reasons, but the paramount one should be striving for improvement. The main point of open source is being able to modify something that barely works for you into something that fits your needs appropriately. Thus the mutual benefit: when you add a feature or fix a bug in an open source product that affects your work, you make that product better for you, but you are also improving its overall value.
The challenge of testing
Among the activities that make the quality of a software product, testing is perhaps the most visible one. There are other elements, such as policies for coding, internal code reviews, and several organizational issues that will affect the final quality. But testing is a key element in software development. Thus, my company has a huge regression test suite that does a good job of keeping each build free of bugs.
As everyone knows, testing is never enough. Despite all the efforts from the developers and the QA engineers, there is always a bug that escapes their attention and affects the user. Why? Surely because testing everything is impossible, but also because developers and QA engineers are highly trained people and they approach problems from a different perspective than the final users.
Common users take the product for what they need, and just throw at it the commands that will solve their problem, regardless of any other concern that the developers could take into account. This naive behavior is what finds the most appalling bugs.
The challenge, then, is to combine the scientific approach of a QA department with the cleverness of willing contributors, who are able to find bugs that elude most of the professionals.
The lessons of Perl
That's why (finally) I am coming here to ask for advice. My job is not as a pure Perl developer (although Perl is largely used in our testing infrastructure), so this problem is not related to Perl as a language, but with Perl as a community. I was accustomed to testing before using Perl, but it is in the Perl community that I found the most efficient way of testing.
I can see that the Perl community has a very good testing infrastructure. I see how it is organized, how it works, but the reasons why it is so good escape me.
Here are the questions
So, fire on. Any insight on this matter could be valuable.
Thanks in advance.
_ _ _ _ (_|| | |(_|>< _|