Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
PerlMonks - Pack script and dependencies into self-extracting bash script

by jbryan (Acolyte)
on Aug 21, 2012 at 17:34 UTC ( #988804=CUFP: print w/replies, xml ) Need Help??

I was playing with PAR::Packer's pp script - and it's amazing! The only problem I've found is that the files it generates (even the perl script -p option) don't run across architectures - e.g. generate a script with pp -p on my 64 bit, but it won't run (for various reasons) on my 32 bit desktop.

So, I grabbed the script included with Module::ScanDeps module and added routines to copy non-core modules to a temporary tree, throw in the file you give it, tar it all up and add a small bash script to decompress the tar, set the lib path(s) with -Mlib and then execute your original script.

How this differs from pp -P:

In the discussion, below, I finally realized the apparent crux of the problem and the nitch where this script is useful:

Files generated with pp -P or pp -p require PAR to be installed on the destination machine. (Or so it seems - am I wrong on that one?)

My use case is that I want to generate a portable file that doesn't require any dependencies on the other machine other than the core modules of Perl itself (for some version of perl.) Or is that an unreasonable goal? Or a goal already covered by some other script/program I'm as yet unaware of?

Note: This script assumes you will use pure-perl modules (e.g. use Crypt::Blowfish_PP instead of Crypt::Blowfish, etc) so that the resulting output file is as portable as possible.

Feel free to flame or tell me I'm just using pp wrong - this just seems to work for my needs, I'm hoping someone else might also find it useful.

Updated script after initial posting: Changed to use Module::CoreList instead of the core-detection logic originally from - this was changed to fix a problem running this script on an older version of perl on one of my older desktop machines. Fixed it to override version provided from Module::CoreList for version. script follows:
#!/usr/bin/perl # - Pack script and dependencies into self-extracting b +ash script # Read about this script at, # or see perldocs at the end of this file. eval 'exec /usr/bin/perl -S $0 ${1+"$@"}' if 0; # not running under some shell $VERSION = '0.80'; use warnings; use strict; use Config; use Getopt::Std; use Module::ScanDeps; use Module::CoreList; use ExtUtils::MakeMaker; use Data::Dumper; use File::Copy qw/copy/; use File::Path qw/mkpath rmtree/; #make_path remove_tree/; use subs qw( _name ); use Data::Dumper; my %opts; #getopts('BVRxce:C:', \%opts); getopts('nxco:C:', \%opts); my (%map, %skip); my $core = 0; #$opts{B}; my $verbose = 0; #$opts{V}; #my $eval = $opts{e}; my $recurse = 1; #$opts{R} ? 0 : 1; # if ($eval) { # require File::Temp; # my ($fh, $filename) = File::Temp::tempfile( UNLINK => 1 ); # print $fh $eval, "\n" or die $!; # close $fh; # push @ARGV, $filename; # } die qq{ Usage: $0 [ -x | -c ] [-C CacheFile ] [ -n ] [ -o OutFile ] File $0 scans File with Module::ScanDeps for non-core dependancies, copies found modules (along with the file given) into a payload tree +, then compresses the payload into a tar archive, prepends a self-extractin +g and exeuctable header to the archive, and writes the archive to OutF +ile (also makes OutFile executable.) OutFile defaults to File with '.bin +' added. Optional Arguments: -o OutFile - Write outut to OutFile instead of the default (File.b +in) -x - Execute code (passes 'execute => 1' to scan_deps()) -c - Compile code (passes 'compile => 1' to scan_deps()) -C CacheFile - Passes CacheFile as 'cache_file => CacheFile' to scan +_deps()) -n - Don't remove stub file/payload tar from cwd on exit } unless @ARGV; my @files = @ARGV; while (<>) { next unless /^package\s+([\w:]+)/; $skip{$1}++; } if(@files > 1) { die "$0 only handles one file argument currently, sorry!"; } my $input = shift @files; # Parse the path and file from the input given my ($input_path,$input_file) = $input =~ /(^.*\/)?([^\/]+)$/; $input_path ||= ''; # prevent warnings of uninitalized value $input_path =~ s/\.\///g; # Strip ext my $input_noext = $input_file; $input_noext =~ s/\..*$//g; # Get the output from command line, if none given, guess my $output = $opts{o} || ''; $output = "${input_noext}.bin" if !$output || $output eq ""; my $output_noext = $input_noext; # Make the payload folder to contain the data my $payload_dir = "$output_noext.files"; if(-d $payload_dir) { warn "[Warn] '$payload_dir' payload folder exists, remove to ensur +e a clean payload."; } mkdir($payload_dir) if !-d $payload_dir; print "$0: Scanning $input for dependencies...\n"; my $map = scan_deps( files => [$input], recurse => $recurse, $opts{x} ? ( execute => 1 ) : $opts{c} ? ( compile => 1 ) : (), #$opts{V} ? ( warn_missing => 1 ) : (), warn_missing => 1, $opts{C} ? ( cache_file => $opts{C}) : (), ); print "$0: Processing dependencies extracted from $input...\n"; my %mods_use; my %noncore_mod_info; my $target_core_ver = 5.05; # Module::CoreList was reporting 'version' was included as of 5.09, bu +t CPAN[1] says version objects added in 5.10 # [1] my %core_ver_overrides = ( 'version' => 5.10 ); # my %fix_dependencies = ( # 'version' => 'version/' # ); foreach my $key (sort keys %$map) { my $mod = $map->{$key}; my $name = $mod->{name} = _name($key); print "# $key [$mod->{type}]\n" if $verbose; if ($mod->{type} eq 'shared') { $key =~ s!auto/!!; $key =~ s!/[^/]+$!!; $key =~ s!/!::!; $mod->{is_bin} = 1; } next unless $mod->{type} eq 'module'; next if $skip{$name}; my $module_core_ver = $core_ver_overrides{$name} || Module::CoreLi +st->first_release($name) || -1; my $is_core = $module_core_ver && $module_core_ver > -1 && $module_core_ver < $target_core_ver ? 1:0; #die "[$is_core] $name: $module_core_ver [target: $target_core_ver +]\n" if $name eq 'version'; if(!$is_core) { my @used_by = @{$mod->{used_by} || []}; $noncore_mod_info{$mod->{key}} = $mod; foreach my $file_key (@used_by) { push @{$mods_use{$file_key}}, $mod->{key}; } } } #die Dumper \%mods_use, \%noncore_mod_info; print "$0: Building payload in $payload_dir\n"; # Routine to copy a given file to the payload folder (creating the des +t folder structure if it doesnt exist) sub add_payload { my $file = shift; my ($path,$name) = $file =~ /(^.*\/)?([^\/]+)$/; my $out_file = $file; $out_file = '/'.$file if $file !~ /^\//; $path ||= ''; # prevent warnings of uninitalized value my $payload_path = $payload_dir.$path; #system("mkdir -p ${payload_dir}${path}") if ! -d "${payload_dir}$ +{path}"; #make_path($payload_path) if !-d $payload_path; mkpath($payload_path) if !-d $payload_path; #system("cp $file ${payload_dir}${out_file}"); copy($file, $payload_dir.$out_file); return wantarray ? ($path,$name) : $file; } # Add the input executable to the payload add_payload($input); # Add all the non-core modules referenced by the script to the payload my @libs; sub contains { my $x=shift; my $ref = shift; foreach my $a(@$ref){retu +rn 1 if $a eq $x; } return 0} my %seen; # Recursively add in dependancies sub add_dependencies { my $file_key = shift; my @noncore_deps = @{$mods_use{$file_key} || []}; foreach my $dep_file_key (@noncore_deps) { #print "\t[debug] add_dependencies('$file_key') => '$dep_file_ +key'\n"; if(add_module($dep_file_key, $file_key)) { add_dependencies($dep_file_key); } } } sub add_module { my $file_key = shift; return 0 if $seen{$file_key}; $seen{$file_key} ++; my $other_key = shift; # just for output my $mod = $noncore_mod_info{$file_key} || die "Invalid file key: ' +$file_key'"; my $file = $mod->{file}; my $name = $mod->{name}; print "$0: Adding non-core dependency '$name' (used by $other_key) +\n"; #from $file\n"; my @dyna_patterns = ('use XSLoader', 'require DynaLoader', 'use Dy +naLoader'); # Ignorant here, is 'use DynaLoader' ever done? my $dyna_count = 0; $dyna_count += 0 + `grep '${_}' $file | wc -l` foreach @dyna_pat +terns; #if($mod->{is_bin}) # doesn't work, {is_bin} never set correctly if($dyna_count > 0) { warn "[Warn] Module '$name' looks it's not Pure-Perl - it prob +ably won't work bundled.\n\tCheck CPAN for a pure-perl version of $na +me to make sure the bundled app works on all architectures"; } my ($file_path,$file_name) = add_payload($file); #print "$path: $name\n"; my @lib_path = split /\//, $file_path; my @mod_path = split /::/, $name; pop @mod_path; # last element of mod is the module itself, e.x. Te +st::Bar would equal Test/ - we just want the folder path, not t +he file # Assuming lib_path is [usr,lib,perl5,HTML,Server] and mod_path is + [HTML,Server], # then we want to pop off the end elements till we reach the 'perl +5' element in lib_path # (the element where they both differ.) Then we throw perl5 back o +n the end of lib_path # and use that as the root of the library tree # This construct causes warnings of uninitalized values #my $tmp = 0; #1 while pop @mod_path eq ($tmp = pop @lib_path); #push @lib_path, $tmp; my $done = 0; while(!$done) { my $p1 = pop @mod_path || ''; my $p2 = pop @lib_path || ''; if($p1 ne $p2) { $done = 1; push @lib_path, $p2; } } my $lib_path = join('/', @lib_path); push @libs, $lib_path if !contains($lib_path, \@libs); #print "copy($file, \"${payload_dir}${file}\")\n"; #print Dumper $mod; return 1; } # Traverse the tree of dependancies, adding them to the payload add_dependencies($input_file); # Generate the stub loader to decompress the payload my $stub = "$output_noext.stub"; my $tmp_dir = "/tmp"; print "$0: Writing stub loader to '$stub'\n"; # Create the lib path used in the stub my $libs = join "\\\n\t", map { s/\/$//g; "-Mlib='$tmp_dir/$payload_di +r${_}'" } @libs; $libs = " \\\n\t$libs \\\n\t" if @libs; # Write the stub itself open(STUB,">$stub") || die "Cannot write $stub"; print STUB qq{#!/bin/bash # Set the extraction location export TMPDIR=$tmp_dir #`mktemp -d /tmp/selfextract.XXXXXX` # Find the line on which the payload starts ARCHIVE=`awk '/^__ARCHIVE_BELOW__/ {print NR + 1; exit 0; }' \$0` # Extract the payload to our folder tail -n+\$ARCHIVE \$0 | tar x -C \$TMPDIR # Set the library path and executable name for perl then jump the the +entry point script perl $libs -e "\\\$0='\$0'; do '\$TMPDIR/${payload_dir}${input_path}/$ +input_file'; die \\\$@ if \\\$@" \$* # Cleanup and exit rm -rf \$TMPDIR/$payload_dir exit 0 __ARCHIVE_BELOW__ }; close(STUB); # Compress the payload into a single archive my $payload_tar="$payload_dir.tar"; print "$0: Compressing $payload_dir to '$payload_tar'\n"; system("tar cf $payload_tar $payload_dir"); # Combine the stub and the payload archive into a single executable if(-f $payload_tar) { system("cat $stub $payload_tar > $output"); system("chmod +x $output"); } else { print STDERR "$payload_tar does not exist\n"; exit(1); } # Cleanup unless($opts{n}) { print "$0: Cleaning up $stub\n"; unlink($stub); print "$0: Cleaning up $payload_dir\n"; unlink($payload_tar); } print "$0: Removing temporary payload tree '$payload_dir'\n"; #remove_tree($payload_dir); rmtree($payload_dir); # Let the user know print "$0: Built $output\n"; sub _name { my $str = shift; $str =~ s!/!::!g; $str =~ s!.pm$!!i; $str =~ s!^auto::(.+)::.*!$1!; return $str; } 1; __END__ =head1 NAME - Build a packed bash script containing a given file an +d it's non-core prerequisites =head1 SYNOPSIS % # Build 'test.bin' from & + dependencies % -o test # Build 'test' from % -n # Build 'test.bin', don't remove +test.stub or test.files.tar =head1 DESCRIPTION F<> is a derivative of the F<> utility included with C<Module::ScanDepps>. Instead of printing out the prerequisites needed by the given file, F<> creates a self-extracting Bash script with the prereqs and the file given that can be distributed to other computers that may lack the non-core prerequisites required. Modules that has loadable shared object files (usually needing a compiler to install) are included, however a warning is printed. Note that this script is not smart enough (yet) to include the shared object files - so likely the resulting file will fail to execute on systems without the module already installed. Your best bet is to find a pure-perl version of the warned module and change your code to use that instead - that is, if your goal is to create a purely portable file. =head1 NOTE This is NOT intended to be a replacement for PAR::Packer, pp, or any of it's ilk. This really just handles a small nitch case, and its a much dumber set of routines than PAR::Packer/pp. The primary use case that I created this script to cover is to create a cross-platform (64bit/32bit) file that only includes non-core modules which will run on systems that already have some 'recent' version of Perl. I ran into problems trying to create a single-file perl script with pp that crossed platforms (32bit to 64bit or vis a versa) which no amount of googling solved. (Mainly having to do with DynaLoader code calls even though I only depended on Pure Perl modules in my code.) In summary, this script works for pure-perl non-core dependancies on systems with perl already installed. It's not a replacement for pp, rather a small nitch script that may or may not work better for very small nitch cases. =head1 OPTIONS =over 4 =item -o OUTPUT_FILE Write the output to I<OUTPUT_FILE> instead of the default (input file name with '.bin' appended) =item -c Compiles the code and inspects its C<%INC>, in addition to static scan +ning. =item -x Executes the code and inspects its C<%INC>, in addition to static scan +ning. =item -n Don't clean up the stub or tar archive created before exiting. =item -C CACHEFILE Use CACHEFILE to speed up the scanning process by caching dependencies +. Creates CACHEFILE if it does not exist yet. =back =head1 SEE ALSO L<Module::ScanDeps>, L<CPANPLUS::Backend>, L<PAR>, L<PAR::Packer> =head1 AUTHORS Josiah Bryan E<lt>josiahbryan@gmail.comE<gt> - Added archive and stub +file creation Audrey Tang E<lt>autrijus@autrijus.orgE<gt> - Original scr +ipt =head1 COPYRIGHT Copyright 2012 by Josiah Bryan E<lt>josiahbryan@gmail.comE<gt> Original script: Copyright 2003, 2004, 2005, 2006 by Audrey Tang E<lt>autrijus@autrijus +.orgE<gt>. This program is free software; you can redistribute it and/or modify i +t under the same terms as Perl itself. See L<> =cut

Replies are listed 'Best First'.
Re: - Pack script and dependencies into self-extracting bash script
by Anonymous Monk on Aug 21, 2012 at 19:35 UTC

    Congratulations, you've reinvented shar/ fatpack

    Congratulations, pp --perlscript can almost do that too :) provided you use -X coremod to exclude core modules

    system qw{ pp -M Data::Dump::FilterContext -M Data::Dump::Filtered -M Data::Dump -X File::Glob -X threads -X Symbol -X subs -X overload -X Exporter -X vars -X utf8 -X strict -X base -X Carp -X Config -X Exporter::Heavy -X feature -X XSLoader -X DynaLoader -X Scalar::Util -X mro -X warnings::register -X integer -X warnings -X MIME::Base64 -X Text::ParseWords -X List::Util -C CACHEFILE --perlscript -o ddvars -e } => 'use Data::Dump; dd\@INC,\%ENV; ';

    Sure various lib/unicore/lib/IDC/ get packed as well, but no binary components get packed, no .dll's or .so's

      So here is my version :)

      #!/usr/bin/perl -- use strict; use warnings; use Data::Dump; use Module::Corelist; @ARGV or die "Usage: $0 outfile infile infile infile\n"; my $outfile = shift || 'ddname'; my @script = @ARGV ? @ARGV : qw[ ]; my @cache = qw[ -C CACHEFILE ]; my %exclude; # we don't pack core modules available in this perl my %include; for ( qx[scandeps @cache -B -V @script ] ){ /^'([^']+)'/ or next; my $mod = $1; my $first = Module::CoreList->first_release($mod); my $removed = Module::CoreList->removed_from($mod); $exclude{$mod}++ if defined $first and $first < $] ; # core in + this perl $include{$mod}++ if defined $removed and $] >= $removed; # removed + (not core) in this perl } my @exclude = map {; '-X', $_ } keys %exclude; my @include = map {; '-M', $_ } keys %include; #~ my @args = ( qw[ pp --perlscript --output ] => $outfile, @exclude, +@include, @script ); my @args = ( qw[ pp -P -o ] => $outfile, @exclude, @include, @script ) +; dd \@args; system @args; __END__

        Impressive coding and deft manipulation - nicely done. I downloaded it and tweaked it to actually run (a typo and Dump->Dumper, print Dumper instead of dd, etc) - nicely done indeed.

        However, the resulting file from PAR still gives an error when run on another machine:

        Can't locate PAR/ in @INC (@INC contains: ...snip...) at /usr/bin/ line 348.

        I even attempted to force inclusion of PAR::Heavy by adding push @include, qw(-M PAR::Heavy); after my @include (Admittedly, I gave up after a few moments of it not working - I didn't try *too* hard) - unfortunately, that didn't change the error message on the other machine at all.

      Ahh, so is that different than what the following does:

      # From % pp -P -o file % pp -B -p -o file

      Neither of the resulting files would run cross-platform (e.g. when copied from 64bit laptop to 64bit machine, both running CentOS, or the other way around - 32bit to 64bit.) Even though no .so/etc were copied, the header on the file that PAR added require'd DynaLoader, which caused it to fail with an error like Undefined subroutine &DynaLoader::bootstrap called at /usr/lib64/perl5/ line 111. (This is from a file generated with pp on my 64bit laptop last night and ran on my 32bit desktop here at the office.)

      I spent several hours googling and trying various options, though I admit I didn't try that -X option (must have glossed over that thru my sleep-deprived hazy vision!)

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: CUFP [id://988804]
Front-paged by Arunbear
[haukex]: Discipulus: which one do you mean?

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (8)
As of 2018-06-22 08:09 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (122 votes). Check out past polls.