Trust, But Verify

[tim.bunce]

Dear layweb, I'm researching for a talk and want to have upto date figures for the size of the test suites in various languages. Especially perl5, parrot, perl6, python, php, and ruby.

I came across chromatic's excellent article and comments. It's a little dated now though.

I was wondering if any monks familiar with one or more of perl5, parrot, perl6, python, php, or ruby, could repeat chromatic's analysis with the latest versions. Ideally distinguishing between the core language and the library.

Thanks!

[chromatic]

Parrot trunk as of today has 11086 tests in 557 files in the default runcore on the most lenient platform I own. We run most of those tests (7133) several times, once for each of seven runcores, so let's say that's 53,884 tests if you run a full test run. Count whichever number you prefer; Python 2.x runs its test suite twice, once against standard textual .py files and once against the compiled/cached .pyc files.

Rakudo by default right now runs 223 tests in 27 files. All pass. If you let it fetch several of the tests from the Pugs repository, 657 tests run in 75 files. Several fail; many files fail to parse.

[moritz]

You can get an overview over number of pugs tests at the smoke server.

The numbers vary, because some commits break parsing of some test files, but you can extract a rough number of about 19k tests that are in the repository atm, and can be parsed by pugs. (Look at "repository snapshot / linux", which should contain the most up to date smoke reports).

Note that these tests are being transformed into the "official" Perl 6 test suite. And TimToady wants twice as many -)

[tim.bunce]

I finished the talk I was working on. The slides and video can be found at http://blog.timbunce.org/2008/03/08/perl-myths/

In the end I didn't include stats of other languages in the slides. I just quote the stats for Perl's own test suite over the last 5 years (slide 44):

2002: Perl 5.8.0 had 26,725 core tests, +41,666 more for bundled libraries etc.

2007: Perl 5.10.0 has 78,883 core tests, +109,427 more for bundled libraries etc.

[djberg96]

You'll want to check the Rubinius test suite. I'm not sure what they're up to lately, but I'm guessing over 10k at this point.

JRuby is using a combination of the Rubinius specs, my own personal, unpublished test suite called 'ruby_test' (at just over 7k tests at this point), BFTS (basically abandoned, the authors are now part of the Rubinius core team), and whatever the JRuby core team decides to add. It may seem a strange mix, but it's actually beneficial, as each group seems to have a different testing style that's mutually beneficial.

You might also want to talk about maintainability because Ho. Ly. God. the Perl test suite is precisely the sort of thing people are talking about when they refer to "the big ball of mud".

[chromatic]

... the Perl test suite is precisely the sort of thing people are talking about when they refer to "the big ball of mud".

The whole thing? Really? Every last bit?

I can think of specific points of awfulness (File::Find is one horrifying example, which shouldn't surprise anyone considering the internals of the module), but it seems rather difficult to argue that the thing is horrible and unmaintainable when it's working.