Re^2: Test Survey Results

[cog]

in reply to Re: Test Survey Results
in thread Test Survey Results

:; time make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_h
+arness(0, 'blib/lib', 'blib/arch')" t/*.t
t/01-conversion_0_10000................ok           
t/02-conversion_10001_999999...........ok       
t/03-conversion_1000000_and_up.........ok     
t/04-special-cases.....................ok   
t/71-version_numbers_are_the_same......ok   
t/pod-coverage.........................ok   
t/pod..................................ok   
All tests successful.
Files=7, Tests=10218,  7 wallclock secs ( 0.94 usr  0.15 sys +  6.70 c
+usr  0.14 csys =  7.93 CPU)
Result: PASS

real    0m7.180s
user    0m7.726s
sys    0m0.324s

There you go, over 10k tests in just 7 seconds.

Granted, that is not a full system, just a module. However, my answers remain: my largest test suite has over 10000 tests, and it runs in just a few seconds.

I also recall creating a system with ~400 tests that would run in just no time. Then again, the most enforced requisite of that platform was that it had to process over 1000 requests per second.

And it did.

[ambs]

Well...

ambs@rachmaninoff ProjectoDicionario$ time make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0)" t/*.t
t/empty_lines....ok                                                          
t/last_line......ok                                                          
t/test_terms.....ok                                                          
All tests successful.
Files=3, Tests=67546, 16 wallclock secs (13.43 cusr +  1.40 csys = 14.83 CPU)

real	0m15.786s
user	0m18.537s
sys	0m1.806s

This is not to show tests run quickly, as the tests are quite simple. What I would like to say is that these tests in particular are not testing code, but testing data quality (a PT dictionary formatting).

Why is this helpful? Because when I too the time to write the tests, I found out that I had more than 6000 entries with formatting errors.

[Anonymous Monk]

Obviously one can have a ton of trivial tests that run in very little time. That doesn't demonstrate a lack of bias. The bias was not contemplating the possibility that an extensive test suite could take much more than 20 minutes to run. In my experience, extensive test suites on a company-wide scale usually take quite a bit more than 20 minutes to run, so I don't consider such to be an 'outlier' as characterized by Ovid to dismiss the criticism. But I'd not have set the upper limit on number of tests at 10,000 either. I'm sure we each have over 100,000 tests at my current and previous employer.