Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Bulk check for successful compilation

by davebaker (Pilgrim)
on Jul 02, 2022 at 21:16 UTC ( #11145248=CUFP: print w/replies, xml ) Need Help??

Just a note to say how much fun it was for me to try the Test::Compile::Internal module, which zips through every Perl module and script in my cgi-bin directory and its subdirectories, making sure each such file successfully compiles.

This lets me feel more at ease about there not being any lurking problems that have arisen due to my having renamed or deleted some custom module, and that scripts or modules I'm still developing haven't "use"d a module and its specified subroutines (whether custom or in my Perl libraries) in a way that misspelled the module name or the subroutine name, or that tries to import a subroutine that doesn't actually exist in the "use"d module (such as a subroutine I meant to add to a "use"d custom module but never got around to adding).

#!/opt/perl524 use strict; use warnings; use Test::Compile::Internal; my $test = Test::Compile::Internal->new(); $test->all_files_ok( '/www/cgi-bin' ); $test->done_testing();

(Edited 7/7/2022 to add hypertext link to metacpan.org page of the Test::Compile::Internal module)

Replies are listed 'Best First'.
Re: Bulk check for successful compilation
by davebaker (Pilgrim) on Jul 05, 2022 at 22:58 UTC

    Aw, neat... this is a new one for me -- how to modify a subroutine in a non-object-oriented CPAN module in a way that doesn't involve monkeying with its actual source code.

    I realized that the Test::Compile::Internal CPAN module doesn't consider ".cgi" files to be Perl "scripts" -- only .pl files, .psgi files, and files that don't have any extension but include the string "perl" in their shebang lines. But I want it to test my .cgi files also (all of which run perl via their shebang lines rather than python or another executable).

    Using brian d. foy's Mastering Perl technique (2d ed. at page 176), I did this, and it worked the first time. This won't blow most PerlMonks readers away, but it's the first time I've used the technique (rather than copying the entire CPAN module's distribution, renaming it Local::Test::Compile::Internal, modifying its source code, etc.).

    Original script, which doesn't check .cgi files (only checks .pl, .pm, .psgi and certain no-extension files, due to the way Test::Compile::Internal works):

    #!/opt/perl524 use strict; use warnings; use Test::Compile::Internal; my $test = Test::Compile::Internal->new(); $test->all_files_ok( '/www/cgi-bin' ); $test->done_testing();

    New script, which also checks .cgi files:

    #!/opt/perl524 use strict; use warnings; BEGIN { use Test::Compile::Internal; no warnings 'redefine'; *Test::Compile::Internal::all_pl_files = sub { my ( $self, @dirs ) = @_; @dirs = @dirs ? @dirs : _pl_starting_points(); my @pl; for my $file ( $self->_find_files(@dirs) ) { if ( $file =~ /(\.pl|\.psgi|\.cgi)$/i ) # <== Here's my m +odification. { # Files with .pl or .psgi or .cgi extensions are perl +scripts push @pl, $file; } elsif ( $file =~ /(?:^[^.]+$)/ ) { # Files with no extension, but a perl shebang are perl + scripts my $shebang = $self->_read_shebang($file); if ( $shebang =~ m/perl/ ) { push @pl, $file; } } } return @pl; } } ## end BEGIN my $test = Test::Compile::Internal->new(); $test->all_files_ok( '/www/cgi-bin' ); $test->done_testing();

    Here is the subroutine's code in the Test::Compile::Internal module that's effectively overridden by the BEGIN code in my script, above.

    sub all_pl_files { my ($self, @dirs) = @_; @dirs = @dirs ? @dirs : _pl_starting_points(); my @pl; for my $file ( $self->_find_files(@dirs) ) { if ( $file =~ /\.p(?:l|sgi)$/i ) { # Files with .pl or .psgi extensions are perl scripts push @pl, $file; } elsif ( $file =~ /(?:^[^.]+$)/ ) { # Files with no extension, but a perl shebang are perl scr +ipts my $shebang = $self->_read_shebang($file); if ( $shebang =~ m/perl/ ) { push @pl, $file; } } } return @pl; }
Re: Bulk check for successful compilation
by davebaker (Pilgrim) on Jul 07, 2022 at 13:42 UTC

    I guess one caveat I might make about the Test::Compile::Internal module is that it tests for successful compilation by using the perl executable (and its associated set of module libraries) that was specified in the shebang line of the script that called it. I'll call it the "tester script" for purposes of the rest of this post.

    If I have multiple Perl installations, the script being tested (i.e., any of the Perl scripts that the Test::Compile::Module locates in various directories) might have a shebang line that specifies the perl executable (which of course has an associated set of module libraries) that is different from the one on the shebang line of the tester script.

    My goal is to see whether the tested script compiles correctly under the perl executable that's going to be compiling and running it -- the one specified on its shebang line. The tested script might be called in the middle of the night by a cron job, or Apache calls it via CGI, or a user enters its path and name at a Linux command line. Here's where a problem arises: If my tester script uses different perl executable/libraries than the tested script, and a particular module has been installed in the Perl libraries used by the tester script but not in the Perl libraries used by the tested script (because I forgot to install it, for example, which might happen if I've upgraded the Perl installation I'm wanting the tested script to run on), the tester script would cheerfully report that the tested script successfully compiles. But in fact it will fail when it's called in practice. A similar problem would arise if a particular module used by a tested script already is correctly installed but it has NOT been installed in the libraries of the tester script (e.g., a CPAN module that's not part of the core modules in the Perl installation under which the tester script is running and has never been installed manually). In that case, the tester script will report that the tested script failed to compile, when in fact it will fail when it's called in practice.

    Would it make sense for the Test::Compile::Internal's code to be modified so that it scrapes the shebang line of the tested file in order to determine and then run the Perl executable (and its corresponding set of Perl libraries) to do the compilation test? Perhaps in most cases there will be no difference between that shebang line and the tester script's shebang line, but sometimes not.

      I did it. I modified the all_pl_files sub further, to scrape the shebang line and store it into a new hash that I named $Test::Compile::Internal::shebang_of_file -- where the keys are the names of the files being tested, and the values are their shebang lines.

      I needed to override another sub also, where the enhancement is applied, namely to replace the perl executable that's running the tester script with the perl executable that's contained on each particular tested script's shebang line, for purposes of assembling the command that Test::Compile::Internal runs on the tested file.

      Works great!

      In addition to scripts (.pl, .psgi, and .cgi files), Test::Compile::Internal checks Perl modules (.pm files). Of course they have no shebang lines, so I haven't tried to replace the perl executable that is used to test modules. Without a way to map the modules to the particular scripts that use them, one worries that a module that uses one or more OTHER modules might be reported as successfully compiled if those other modules are found in the Perl libraries of the perl executable running the tester script, even though one or more of them is missing from the Perl libraries that actually will be used by the scripts being tested. But pretty much any perl executable should work to find all syntax errors in the modules (which is one reason they wouldn't compile successfully), and the problem of "missing" modules is a problem that would seem to be detected and reported as a compilation failure when Test::Compile::Internal tests the scripts that use them.

      #!/opt/perl524 use strict; use warnings; BEGIN { use Test::Compile::Internal; no warnings 'redefine'; *Test::Compile::Internal::all_pl_files = sub { my ( $self, @dirs ) = @_; @dirs = @dirs ? @dirs : _pl_starting_points(); my @pl; for my $file ( $self->_find_files(@dirs) ) { if ( $file =~ /(\.pl|\.psgi|\.cgi)$/i ) { # Files with .pl or .psgi or .cgi extensions are perl +scripts push @pl, $file; my $shebang = $self->_read_shebang($file); if ( $shebang =~ m/perl/ ) { $shebang =~ s/\s+$//; $Test::Compile::Internal::shebang_of_file{$file} = $shebang; } } elsif ( $file =~ /(?:^[^.]+$)/ ) { # Files with no extension, but a perl shebang are perl + scripts my $shebang = $self->_read_shebang($file); if ( $shebang =~ m/perl/ ) { push @pl, $file; $shebang =~ s/\s+$//; $Test::Compile::Internal::shebang_of_file{$file} = + $shebang; } } } return @pl; }; *Test::Compile::Internal::_perl_file_compiles = sub { my ( $self, $file ) = @_; if ( !-f $file ) { $self->{test}->diag("$file could not be found") if $self->verbose(); return 0; } my @inc = ( 'blib/lib', @INC ); my $taint = $self->_is_in_taint_mode($file); my $command; if ( $Test::Compile::Internal::shebang_of_file{$file} ) { ( my $executable_filename = $Test::Compile::Internal::shebang_of_file{$file} + ) =~ s/#!//; $command = join( " ", ( $executable_filename, "-c$taint", +$file ) ); } else { $command = join( " ", ( qq{"$^X"}, ( map {qq{"-I$_"}} @inc ), "-c$taint", $f +ile ) ); } if ( $self->verbose() ) { $self->{test}->diag( "Executing: " . $command ); } my ( $compiles, $output ); eval { ( $compiles, $output ) = $self->_run_command($command); # $output is a reference to an array of one or more lines. 1; } or do { my $array_ref_holding_custom_output = ["Can't run command '$command' for compilation test +on file '$file': $@"]; ( $compiles, $output ) = ( 0, $array_ref_holding_custom_ou +tput ); }; if ( $output && ( !defined( $self->verbose() ) || $self->verbose() != 0 + ) ) { if ( !$compiles || $self->verbose() ) { for my $line (@$output) { $self->{test}->diag($line); } } } return $compiles; }; } ## end BEGIN my $test = Test::Compile::Internal->new(); $test->all_files_ok('/www/cgi-bin'); $test->done_testing();

      My BEGIN code needed to override a second subroutine, as shown above, called _perl_file_compiles. Here is the subroutine's code in the Test::Compile::Internal module:

      sub _perl_file_compiles { my ($self, $file) = @_; if ( ! -f $file ) { $self->{test}->diag("$file could not be found") if $self->verb +ose(); return 0; } my @inc = ('blib/lib', @INC); my $taint = $self->_is_in_taint_mode($file); my $command = join(" ", (qq{"$^X"}, (map { qq{"-I$_"} } @inc), "-c +$taint", $file)); if ( $self->verbose() ) { $self->{test}->diag("Executing: " . $command); } my ($compiles, $output) = $self->_run_command($command); if ( $output && (!defined($self->verbose()) || $self->verbose() != + 0) ) { if ( !$compiles || $self->verbose() ) { for my $line ( @$output ) { $self->{test}->diag($line); } } } return $compiles; }

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: CUFP [id://11145248]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (4)
As of 2022-08-13 22:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?