Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

And now write 100 times: "I'll automate our test-environment"

by Tomte (Priest)
on May 12, 2004 at 10:35 UTC ( #352687=perlmeditation: print w/ replies, xml ) Need Help??

Once again one of my stupid "I overlooked that" bugs made me and my company look bad in the eyes of a customer; and me feel miserable, because I do know better than that...

I quickly implemented a change to our CMS, allowing our customer to manually affect a certain behaviour, if it (the customer ;) wishes to do so. As to not effect other customers, this allowance can be switched on and off.

Alas, if switched off, information was deleted due to an Exception thrown in absence of the optional parameter.

The problem is I only thought I tested all cases; I obviously didn't (falling sick doesn't help focus, but is a weak excuse). My colleague, who put my changes onto the live server didn't tested all cases either; trusting me or being lazy or probably both, who knows.

I tried here and there to introduce unit- and integration-tests to our development-procedures and failed miserably. This incident did provide another reason and another proof that I nevertheless have to automate our test-environment...

...or become butcher or carpenter

to fix it and put it online took minutes, but I'm still feeling kind of unprofessional,
tomte


An intellectual is someone whose mind watches itself.
-- Albert Camus

Update: (2004/05/13) Thanks for your input and consolation; it is greatly appriciated.

After a good nights sleep, the world looks bit better than yesterday; and there's a meeting scheduled for next week, to see what we all can agree upon to change in our development-method to gain an enhanced quality output without sacrificing too much speed (in total reaction-/release-time) or to increase needed effort too much. Maybe this time...

Comment on And now write 100 times: "I'll automate our test-environment"
Re: And now write 100 times: "I'll automate our test-environment"
by tachyon (Chancellor) on May 12, 2004 at 11:06 UTC

      ++tachyon. I bookmarked the program; alas it won't help me at work, as most of the stuff here is coded in java

      regards,
      tomte


      An intellectual is someone whose mind watches itself.
      -- Albert Camus

        use JUnit, perhaps? I've heard rave reviews about it.

        You could implement a similar framework for Java testing in Perl using Inline::Java within the test scripts to access the java classes (use STUDY or AUTOSTUDY options) Then you get all the power and simplicity of the Perl testing framework but can run the tests against Java Classes. You could also write some tests in Java and some in Perl as you desire. Given the far more strongly typed nature of Java you could stub up all sorts of 'standard' tests automatically. Just a thought.

        cheers

        tachyon

Re: And now write 100 times: "I'll automate our test-environment"
by Steve_p (Priest) on May 12, 2004 at 11:41 UTC

    Well, like most bad experiences, maybe you can get something good to come from this. The main problem appears to have been that you didn't know exactly what you have tested. So, rather than feeling miserable, head to your nearest CPAN and check out Devel::Cover. You'll at least have the solace in knowing that test coverage shouldn't cause you problems again.

Re: And now write 100 times: "I'll automate our test-environment"
by nimdokk (Vicar) on May 12, 2004 at 12:11 UTC
    I feel your pain, I went through it just last week. Part of the resolution was much stricter processes and procedures for stepping through the SDLC environments. The team I work on does not have a tiered development environment but one of the groups we work with does, so we have to approximate it as much as possible. Fortunately, the system is young enough that we can make changes for the better.
Re: And now write 100 times: "I'll automate our test-environment"
by flyingmoose (Priest) on May 12, 2004 at 19:27 UTC
    Automation can never hurt, but it is never a replacement for real human testing. And you can easily get into situations that are not cleanly testable -- scalability of large distributed apps, machine dependancy, kernel dependancy, timing issues only seen in real world apps, whitebox mentality of end users, firmware or driver involvement in some cases, etc, can result in test automation that requires a huge farm of available hardware. It depends on what kind of code you write, I guess, but in my case (at least), I see a lot of code (not just Perl) that does not fit cleanly into automated test frameworks. Moral of the story -- know when to use automated testing, as well as when automated testing is not the only solution. Obviously having an organization that doesn't agree to spend cycles in Q/A is a bad one in need of fixing!

      Automation can never hurt, but it is never a replacement for real human testing.
      As a general statement: you're of course right, but automatic testing is a real help in focusing your manual testing on broken things while fixing them. If we had a test-suite for the part I enhanced, I had noticed I broke it, and had a clear indicator as to where to look; we have automated builds in place, that inform you of errors/success in building a clean, freshly checked out version after you checked in new code (builds are done every ten minutes, if necessary); we should strive to do testing there too.

      It depends on what kind of code you write, I guess, but in my case (at least), I see a lot of code (not just Perl) that does not fit cleanly into automated test frameworks.
      This is the case here too with nearly all "legacy code"; you write different code, if you write with automated tests in mind; but I think this is a GoodThing(tm) -- and we should at least try to get to a point, where newly written code could be tested...

      know when to use automated testing, as well as when automated testing is not the only solution. Obviously having an organization that doesn't agree to spend cycles in Q/A is a bad one in need of fixing!
      Obvoulsy right :)
      See my update to the OP; we will see if we can agree to something to assure more quality...

      regards,
      tomte


      An intellectual is someone whose mind watches itself.
      -- Albert Camus

        Perhaps you think this is a "legacy" issue, but it's not. Write some more systems-oriented code (not neccessarily simple Perl scripts or even Perl applications) and you will see what I mean. Anyhow, if you don't understand my caution, it doesn't apply to you (yet)... which is ok.

        As for running builds as part of the test suite, umh, the developers had better run the builds themselves, else you have a serious issue of incompetance to deal with if they think they can check in uncompiled code! At least nightly builds are a very good thing, of course.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://352687]
Approved by adrianh
Front-paged by ChemBoy
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (3)
As of 2014-09-22 02:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (177 votes), past polls