Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked

manpages file naming - a filesystem difference dilemma

by Intrepid (Deacon)
on Nov 22, 2006 at 14:25 UTC ( #585526=perlquestion: print w/ replies, xml ) Need Help??
Intrepid has asked for the wisdom of the Perl Monks concerning the following question:

The scenario, in brief:

One has a removable media device like a USB (thumb|key)drive and one wants to carry a bunch of Perl modules around on it, mounting it to a workstation as needed, and including the path to the modules in @INC (via the $PERL5LIB mechanism). This could happen on a UNIX workstation or a MS Windows one, it does not matter.

Installing modules to this USB/removable (thumb|key)drive in the normal manner involves setting a PREFIX= and possibly LIB= parameters for ExtUtils::MakeMaker -based module build-infrastructure supported modules, or the parallel means for a Module::Build -based one. Then you just go do the make ; make test ; make install or Module::Build analog as you would for any module installation. I guess for some readers this is going to sound like an exotic thing to envision, but for the author this is not. I've been doing things like this with the manual installation of modules for years. I understand all the mechanisms and gotchas involved, and IMHO the support for this in Perl is very good and the results I get are very satisfying.

The Problem:

When the build host is UNIX (i.e. GNU/Linux) the make process will create man pages for all detected *.pm *.pl and *.pod files. This is "normal" for the architecture / os type and I like and want the manpages to be installed to the file hierarchy on the removable media drive. However, the filenames of man files produced from the POD in modules are like Module::Kewl::Excelsior.pm3 on a UNIX system. Note the double-colons (::).

The filesystem of a USB (thumb|key)drive is (nearly ?) invariably FAT32 as shipped by the manufacturer (and for my purposes, for maximum portability, needs to be kept that way). A FAT-ish filesystem, having originated from MS DOS, does not accept colons : as part of filenames. They are (obviously) barred from the set of ascii characters allowed because of the special semantic meaning they have on an MS DOS-ish / Windows-ish system. The result is that the manpages cannot be written to the removable FAT/FAT32 media at make install time, possibly generating error messages or possibly just failing silently (the error message will be the ubiquitous and vastly informative MS Windows error No such file or directory).


I like manpages. Manpages are good to me. They give me a feeling of reassurance that I have the proper up to date documentation (assuming the module dist maintainer is keeping the documentation in sync with the changes ;-) for the code I am using. Bare-bones Debian and Debian-based systems don't ship with a perldoc command (this is old news ...), so that's an instance where having manpages built is substantially important to having the POD accessible to me as the user. This is about personal preference anyway. I'll downvote replies that consist of "So use perldoc instead" and / or "So don't install / live without the manpages" (go ahead, downvote this node for mentioning downvoting (or because Intrepid authored it); you know you want to; why try to change now?).

Discussing solutions:

On Cygwin, which is UNIX but lives within the special needs of the MS Win filesystem semantics WRT special characters and filenames (AUX, PRN, NUL, etc), the manpages are usable and are built. The ExtUtils::MM_Cygwin module overrides the method which generates make recipes to build manpages from module or script files; it provides a . (dot) in place of the :: (double-colon) in creating the filenames for the man pages that will be copied into place when everything installed.

A solution like the one used by Cygwin(perl) is great and is what I think I want, but it requires one to make some change to the files included in the module distribution; either some hand-editing or some addition of files or both. Can anyone think of a way to achieve this manipulation of EU::MM (or M::B) operations that doesn't require that kind of a tedious manual operation for each module to be built/installed to the removable media drive? Or does anyone have a suggestion for a new method or parameter or functionality for EU::MM or M::B that would allow for changing the manpage filename namespace delimiter to be compatible with FAT filename semantics? I can envision for example a $PERL_MM_USE_MANFILE_SEP=. environ var that if present would trigger the desired overidding.

    Soren A / somian / perlspinr / Intrepid

Words can be slippery, so consider who speaks as well as what is said; know as much as you can about the total context of the speaker's participation in a forum over time, before deciding that you fully comprehend the intention behind those words; if in doubt, ask for clarification before you 'flame'.

Comment on manpages file naming - a filesystem difference dilemma
Select or Download Code
Replies are listed 'Best First'.
Re: manpages file naming - a filesystem difference dilemma
by shmem (Canon) on Nov 22, 2006 at 15:17 UTC
    As a quick dirty hack, I'd copy over ExtUtils/ into ~/privdir/ExtUtils and hack sub install to apply a s/::/./g on $targetfile if that file happens to be a manual page. Then, if I wanted to install onto $thumbdrive, I'd just prepend ~/privdir to PERL5LIB and done.


    _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                  /\_¯/(q    /
    ----------------------------  \__(m.====·.(_("always off the crowd"))."·
    ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

      A great reply, shmem, IMHO

      Here's my present workaround based on shmem's suggestion above. Only the install sub from ExtUtils::Install is shown (it is the only place in the module file where (at present) any changes have been made). I'll post a link to a network URI where a patch can be fetched in a later update.

      sub install { my($from_to,$verbose,$nonono,$inc_uninstall) = @_; $verbose ||= 0; $nonono ||= 0; my $bootstrapped = eval 'use Filesys::Type 0.02 (qw|fstype|); 1;'; + # CPAN my $no_colons_in_basenames; my @DOSish_FSTs = qw(msdos umsdos vfat ntfs iso9660 smb FAT FAT32 +CDFS NTFS); =for COMMENTARY # Types of fs that can be returned by Filesys::Type::fstype would +have been nice # to have access to without breaking into the module's encapsula +tion. (this is IMHO # nonoptimal design; these could/should have been exportable fro +m the module). =cut use Cwd qw(cwd); use ExtUtils::Packlist; use File::Basename qw(dirname); use File::Copy qw(copy); use File::Find qw(find); use File::Path qw(mkpath); use File::Compare qw(compare); my(%from_to) = %$from_to; my(%pack, $dir, $warn_permissions); my($packlist) = ExtUtils::Packlist->new(); # -w doesn't work reliably on FAT dirs # UHH, FAT-type filesystems can be found on other than MSWin32 OS' +s. Huh? XXX $warn_permissions++ if $^O eq 'MSWin32'; local(*DIR); for (qw/read write/) { $pack{$_}=$from_to{$_}; delete $from_to{$_}; } my($source_dir_or_file); foreach $source_dir_or_file (sort keys %from_to) { #Check if there are files, and if yes, look if the corresponding #target directory is writable for us opendir DIR, $source_dir_or_file or next; for (readdir DIR) { next if $_ eq $Curdir || $_ eq $Updir || $_ eq ".exists"; my $targetdir = install_rooted_dir($from_to{$source_dir_or +_file}); ++$no_colons_in_basenames if grep(fstype($targetdir) eq $_ + , @DOSish_FSTs); mkpath($targetdir) unless $nonono; if (!$nonono && !-w $targetdir) { warn "Warning: You do not have permissions to " . "install into $from_to{$source_dir_or_file}" unless $warn_permissions++; } } closedir DIR; } my $tmpfile = install_rooted_file($pack{"read"}); $packlist->read($tmpfile) if (-f $tmpfile); my $cwd = cwd(); MOD_INSTALL: foreach my $source (sort keys %from_to) { #copy the tree to the target directory without altering #timestamp and permission and remember for the .packlist #file. The packlist file contains the absolute paths of the #install locations. AFS users may call this a bug. We'll have #to reconsider how to add the means to satisfy AFS users also. #October 1997: we want to install .pm files into archlib if #there are any files in arch. So we depend on having ./blib/arch #hardcoded here. my $targetroot = install_rooted_dir($from_to{$source}); my $blib_lib = File::Spec->catdir('blib', 'lib'); my $blib_arch = File::Spec->catdir('blib', 'arch'); if ($source eq $blib_lib and exists $from_to{$blib_arch} and directory_not_empty($blib_arch)) { $targetroot = install_rooted_dir($from_to{$blib_arch}); print "Some files found in $blib_arch: we shall therefore +be " . "installing files in $blib_lib into the architecture + dependent " . "library tree\n"; } chdir $source or next; find(sub { my ($mode,$size,$atime,$mtime) = (stat)[2,7,8,9]; return unless -f _; my $origfile = $_; return if $origfile eq ".exists"; my $targetdir = File::Spec->catdir($targetroot, $File::Find:: +dir); my $targetfile = File::Spec->catfile($targetdir, $origfile); my $sourcedir = File::Spec->catdir($source, $File::Find:: +dir); my $sourcefile = File::Spec->catfile($sourcedir, $origfile +); # Cope with installation of man files to FAT type filesyst +ems (could # be installing to removable media formatted as vfat/FAT32 + from a # UNIX OS, like GNU/Linux or Cygwin or *BSD, for example). if($sourcedir =~ m{ blib/man[31] }x and $no_colons_in_base +names) { my $formername = $targetfile; $targetfile =~s{::} {.}g; warn qq|INFO: "$formername"\n => "$targetfile"\n| , qq|for writing to the target location which is a| , (' '.fstype($targetdir)) , qq| filesystem (no colons allowed).\n|; } my $save_cwd = cwd; chdir $cwd; # in case the target is relative # 5.5.3's File::Find missing no_chdir option. my $diff = 0; if ( -f $targetfile && -s _ == $size) { # We have a good chance, we can skip this one $diff = compare($sourcefile, $targetfile); } else { print "$sourcefile differs\n" if $verbose>1; $diff++; } if ($diff){ if (-f $targetfile){ forceunlink($targetfile) unless $nonono; } else { mkpath($targetdir) unless $nonono; print "mkpath($targetdir)\n" if $verbose>1; } copy($sourcefile, $targetfile) unless $nonono; print "Installing $targetfile\n"; utime($atime,$mtime + $Is_VMS,$targetfile) unless $nonono>1; print "utime($atime,$mtime,$targetfile)\n" if $verbose>1; $mode = 0444 | ( $mode & 0111 ? 0111 : 0 ); chmod $mode, $targetfile; print "chmod($mode, $targetfile)\n" if $verbose>1; } else { print "Skipping $targetfile (unchanged)\n" if $verbose; } if (defined $inc_uninstall) { inc_uninstall($sourcefile,$File::Find::dir,$verbose, $inc_uninstall ? 0 : 1); } # Record the full pathname. $packlist->{$targetfile}++; # File::Find can get confused if you chdir in here. chdir $save_cwd; # File::Find seems to always be Unixy except on MacPerl :( }, $Is_MacPerl ? $Curdir : '.' ); chdir($cwd) or Carp::croak("Couldn't chdir to $cwd: $!"); } $no_colons_in_basenames = undef; # XXX ? if ($pack{'write'}) { $dir = install_rooted_dir(dirname($pack{'write'})); mkpath($dir,0,0755) unless $nonono; print "Writing $pack{'write'}\n"; $packlist->write(install_rooted_file($pack{'write'})) unless $nono +no; } }

          Soren A / somian / perlspinr / Intrepid

      Words can be slippery, so consider who speaks as well as what is said; know as much as you can about the total context of the speaker's participation in a forum over time, before deciding that you fully comprehend the intention behind those words. If in doubt, ask for clarification before you 'flame'.

        As I read your initial post, I came up with the same idea as shmem. But after some digging, I think there's an easier solution.

        Manpage separators are provided by replace_manpage_separator() in each EU::MM_* file. But the method is only called from EU::MM_Unix::init_MAN3PODS(). So you might be able to just patch that file to do something specific instead of calling the OS-specific EU::MM_* file.

        - $manpagename = $self->replace_manpage_separator($manpagename); + $manpagename =~ s{/+}{.}g; # from

        The inheritance structure is hard to follow or I'd suggest something more clever than copy/patch.


        Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

        hmm.. with a context diff it would be easier to spot your changes.



        _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                      /\_¯/(q    /
        ----------------------------  \__(m.====·.(_("always off the crowd"))."·
        ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
Re: manpages file naming - a filesystem difference dilemma
by mattr (Curate) on Nov 23, 2006 at 17:21 UTC
    Interesting responses. Seems like one of the above suggestions would be a very nice addition to be recommended to the author.

    Well, my own 2 cents and not so elegant.. I once looked at a tool that would give you a virtual unix filesystem packed into one windows file. Posted it here somewhere.. ah, here I was asking about PAR and gui app bundling tools, and found Thinstall. It's not free and maybe not what you want, that is you might as well use cygwin I would guess then.

    Of course you could also just build to a local folder on unix and then write a small utility to copy while renaming onto your usb stick. That way you don't have to recompile just to get the files renamed. Would this be sufficient? Of course if you actually want to run these on windows then you probably want to compile them for windows if they are not PurePerl but anyway..

    Um, also I don't mean to be silly but I wonder if perhaps you just tar the files up then a windows untar utility might even rename them for you? Apparently this is not uncommon, I looked and found on the net Michael Husted's freeware untar. So maybe you could just tar up your man pages folder and untar it on windows. Well anyway here it is. Of course it would be nice if MM just did what you need but here's the info for this sort of meta-weasling away from the problem.

    Untar v1.14 Description: unpacks TAR (Tape ARchives) files often used in Unix and Linux systems +. Subdirectories and illegal characters in file names (due to differe +nt file name formats between DOS and Unix) can be automatically handl +ed by the program. Untar also supports long file names at the DOS-prompt of Windows 95/98 +/NT by renaming the files after untaring them.
Re: manpages file naming - a filesystem difference dilemma
by Anonymous Monk on Nov 22, 2006 at 18:25 UTC
    Bare-bones Debian and Debian-based systems don't ship with a perldoc command (this is old news ...)
    Since when do bare-bones Windows-based systems ship with a man command? Let alone Cygwin...
Re: manpages file naming - a filesystem difference dilemma
by Anonymous Monk on Nov 22, 2006 at 15:30 UTC
    Bare-bones Debian and Debian-based systems don't ship with a perldoc command (this is old news ...)
    Why don't you ship perldoc?

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://585526]
Approved by blue_cowdawg
Front-paged by liverpole
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (14)
As of 2015-07-28 20:09 GMT
Find Nodes?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...

    Results (258 votes), past polls