cpan test redux...better, but...

[perl-diddler]

The latest test reports on CPAN show up at
    http://www.cpantesters.org/distro/P/P.html.

In there, only 3 people had failures:
    With 1 tester having 1 report on MSWIN on XP in German.
    With 2nd tester having 3 FAILS on perl 5.{8.{8,9},18.1}
        but 11 PASSES on other versions.
    With a 3rd tester, having 100% fails (23/23 FAIL).

Except for the 5.8 version of perl, which might have some
compat prob(s) I need to work around, the rest
are the same error as commented on here:
    http://www.perlmonks.org/?node_id=1061652.

Specifically, these lines that seemingly conflict:

PERL_DL_NONLAZY=1 /home/sand/src/perl/repoperls/installed-perls/perl/v
+5.12.5/9980/bin/perl "-MExtUtils::Command::MM" "-MTest::Harness" "-e"
+ "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/a
+rch')" t/*.t
Can't locate mem.pm in @INC (you may need to install the mem module) (
+@INC contains: [...]:.) at lib/P.pm line 150.
BEGIN failed--compilation aborted at lib/P.pm line 150.
## and
------/PREREQUISITES/------
Prerequisite modules loaded:
requires:
    Module              Need   Have    
    ------------------- ------ --------
    mem                 v0.3.2 0.3.2   
    Test::Simple        0.44   1.001002

build_requires:
    Module              Need   Have    
    ------------------- ------ --------
    mem                 v0.3.2 0.3.2   
----------------------------

This was also a problem in 1.1.1, but there it was near 100%, across the board.

I also had a "use_ok(P)" and use_ok(mem) at the beginning
-- and called BAIL_OUT if they were not found. I
tested the missing cases, and DID get the BAIL_OUT message.

So now I'm getting 'mem' being both present and non-present
in the same test. This doesn't make sense.

I did contact the tester with all the fails and asked if
they could think of anything that might be causing
this... will have to see if they have ideas or even
respond (they might be busy or something)...

Anyway -- just thought I'd throw this out, since this
aspect of the original post managed to not be addressed
(likely due to no one else being a physicist
who could explain how Schrodinger's theories applied to perl testing;
:^| ).
--------------
Got email back from user.

They said it looked like the program under test (being called from the test program), did not have the correct PERL5LIB set in the environment. It seems test programs are expected to copy %INC to PERL5LIB? That seems an odd requirement for a test script???

I.e. a Test harness driven by some script, that calls or invokes program under test with various inputs, and checks for correct output. At least that's how I'm used to testing, since you don't want your scripts to be in the same program as the program under test so the scripts won't be accidently mangled by the program under test. Is that not correct procedure? *wimper*?

[Anonymous Monk]

Re: cpan test redux...better, but... still shouldn't be on CPAN

"mem" is particularly bad name for a pragma, "mem" has nothing to do with memory; markloaded or markused sound better (and like they should be part of Module::Loaded)

P is a particularly bad name for a module, its a single letter top level namespace . So it exports a function of the same name P ( and Pe), doesn't make P a good module name -- If you need it available from a commandline consider your own Toolkit

PAUSE: pause_namingmodules On The Naming of Modules

CPAN Testers Reports: Report for P-1.1.4 shows a different failure mode (bugs in your code)

So now I'm getting 'mem' being both present and non-present in the same test. This doesn't make sense.

Funny, and then what happened?

If your test suite isn't telling you what's going on, improve it

I did contact the tester with all the fails ... and asked if they could think of anything that might be causing this... will have to see if they have ideas or even respond (they might be busy or something)...

When I was a tester, I did not appreciate being contacted like this; cpantesters isn't perlmonks/stackoverflow ... they run your test suite and submit reports automatically, most everything else is up to you; If your tests don't tell you enough about what the problem is, write better tests "Can't CPAN Testers read? I have detailed the format I require for bug reports!"

Just something to keep in mind because excessive "spam" from authors can drive some cpantesters away , its partially why I quit cpantesters

Got email back from user. They said it looked like the program under test ...

I just read this part of your post, well /dev/null isn't portable for starters ...

ddumper^{Basic debugging checklist} Basic debugging checklist item 4 works for debugging even in test suites

Thats all the interest I've managed to muster, I'm fickle lately

[tobyink]

Having a look at your latest release (1.1.5), your test is now a shell script rather than a Perl script. This is really going to FAIL across the board.

Firstly, Windows users are unlikely to have /bin/sh. And secondly, although TAP::Harness can be configured to run shell scripts instead of Perl scripts, your Makefile does not attempt to do so, so the test is being run as if it's Perl.

Is perl Makefile.PL; make; make test even working on your own machine? I can't imagine that it is! If it's not working on your machine, don't expect it to work on anyone else's either!

Really, really, really, you should not need to perform all these manipulations to get your tests to pass!

I'll write back in a few minutes with a patch for 1.1.5 to get tests passing.

UPDATE: here's a patch to fix all the weirdness, and here's a tarball of P-1.1.5 with my fixes applied. If you took that tarball, bumped the version number and released it to CPAN, I imagine you'll get plenty of PASS results from CPAN testers.