Spec test location (was: Re^2: Getting Involved with Perl 6 - 2009)

[moritz]

Thanks for bringing it up, I'll try to summarize the current state:

• The pugs repo is used as a general repository for Perl 6 related projects. Although we still call it "pugs repo", it hosts the specification, test suite, elf, mildew, STD.pm, some infrastructural code (IRC logging, evalbot, ...). We think about renaming the repository one day.
• The test suite won't move into the rakudo repository. It is meant to be implementation agnostic, and moving it into the repo of an implementation would be counter-productive (and it's used by other programs as well, namely STD.pm and elf). Maybe I'll move it out into a completely separate repository one day.

Currently none of these issues are pressing, so I won't hurry. Especially I want to sort out the rest of the old pugs test suite before doing any kind of move. Since the easy-to-verify tests have already been moved, this is tedious work, and going to take some time. I also consider using git instead of SVN, but I'll have to wrap TimToady's head around git first ;-)

Any move will be carefully coordinated with the heads of the others projects, of course.

[kyle]

Thanks for your reply! I hope you don't mind if I have a few more questions.

I find git to be very SVN-like when using Cogito. In fact, I rarely type 'git' at all. (I guess that's more a note for TimToady than you.)

What's involved in sorting out the old Pugs test suite? Off the top of my head, I'd think that it means:

foreach my $file ( @pugs_test_files ) {
    my $new_file = reasonable_spectest_location_for( $file );
    move $file, $new_file;
    add_fudges( to => $new_file, for => $_ )
        for @perl6_implementations;
}

If I were doing this, I think the hard parts would be,

1. Figuring out where to put the new file in the spectest directory, since I'm not very familiar with the spec. (Perhaps this is a way to get familiar.)
2. Fudging anything other than Rakudo or Pugs, since I don't have them.

After that, I'm guessing it's grunt work of running the tests and seeing how they blow up to get them fudged properly for Rakudo. (Skip tests that die, and 'todo' tests that fail.) If I were doing it, I'd probably let the other implementation authors be responsible for their own fudges. Or is that rude?

Is there more involved than that?

[moritz]

The most difficult problem while moving the tests is deciding if they still conform to the current spec.

Some tests use (or test) rather obscure features of which I don't know if they are specced, and if yes, where. The specs are a massive lump of text (more than 31k lines) and rather dense, so that's not always easy to decide. Try to search for the operator but in English text isn't entirely easy, for example.

OTOH fudging isn't really necessary for moving tests into spec/: all implementations that we care about don't expect all test files to pass, only a selection of them. (Pugs does, but since pugs isn't maintained anyway, I don't care for now).

[kyle]

That's harder than I thought. I'd be inclined to assume that passing tests are still correct, but even those might be testing an old spec which has since changed, but the implementations haven't caught up. So one really has to consider every single test individually.

Thanks, that clears that up. I'll put a note about this in the documentation where I would have wanted to find it.