Munging file name, to be safe- & usable enough on Unix-like OSen & FAT32 file system

[parv]

A program written in a hurry some time ago to munge file paths generally for file systems for Unix(-like) OSen & specifically for FAT32.

Learned the hard way that NTFS would allow file names to be written to FAT32 even if some characters are outside of FAT32 specification. Problematic characters seemed to be en- & em-dash, fancy quotes, pipe, Unicode "?", & possibly few others (web pages saved with title as the file name). Mounting FAT32 file system on FreeBSD with specific codepage(s), or "nowin95" or "shortnames" mount options did not help (mount_msdosfs(8)). Munging it was then🤷🏽‍♂️

# quick-sanename.pl

use strict;
use warnings;
use feature qw[ state ];
use File::Copy  qw[ move ];


@ARGV
  or die qq[Give a file name to sanitize.\n];

my $dry_run   = 0;
my $noisy     = 1;
my $lowercase = 1 ;

my $for_windows = 1;
my $clean_past_255 = 1;

# General cleansing of base names.
my %cleansed = run_cleanser( \&Cleanser::cleanse, @ARGV );

if ( $for_windows )
{
   if ( ! %cleansed )
   {
      %cleansed = run_cleanser( \&Cleanser::cleanse_for_windows, @ARGV
+ );
   }
   else
   {
      # Work on the changes of general cleansing.
      while( my ( $old, $once ) = each %cleansed )
      {
         my $again = Cleanser::cleanse_for_windows( $once ) or next;
         $cleansed{ $old } = $again;
      }
      # Take care of those which were skipped during general cleansing
+.
      my @todo = grep { ! exists $cleansed{ $_ } } @ARGV;
      my %win_cleansed = run_cleanser( \&Cleanser::cleanse_for_windows
+, @todo );
      %cleansed = ( %cleansed, %win_cleansed );
   }
}

%cleansed
   or die qq[No new file names were generated.\n];
# Move file.
for my $old ( sort keys %cleansed )
{
   my $new = $cleansed{ $old };

   if ( $noisy || $dry_run )
   {
      printf qq['%s'  ->  '%s'\n] , $old, $new;
   }
   $dry_run and next;
   if ( -e $new )
   {
      warn qq[Skipped rename of "$old", "$new" already exists.\n];
      next;
   }
   if ( ! move( $old, $new ) )
   {
      warn qq[Could not move "$old" to "$new": $!\n];
   }
}

exit;


sub run_cleanser
{
   my ( $clean_type, @old_path ) = @_;
   @old_path or return ();

   my %out;
   for my $one ( @old_path )
   {
      my $new = $clean_type->( $one ) or next;
      $out{ $one } = $new;
   }
   return %out;
}

BEGIN
{
   package Cleanser;

   use File::Basename qw[ fileparse ];
   use File::Spec::Functions qw[ canonpath catfile ];

   sub path_if_diff
   {
      my ( $old, $dir, $cleaned_base ) = @_;

      $lowercase and $cleaned_base = lc $cleaned_base;
      my $new = canonpath( catfile( $dir, $cleaned_base ) );
      return $old ne $new ? $new : undef;
   }

   # Returns a cleaned path if possible; else C<undef>.
   #
   # Substitues various characters with "_" as minimally as possible.
   sub cleanse
   {
      my ( $old_path ) = @_;

      # Yes, I do mean to keep any word & digit in any writing script 
+(language).
      #state $alnum = 'a-zA-Z0-9';
      state $alnum = '\w\d';

      # quotemeta() does not escape "(" which causes warning that it w
+ould be
      # deprecated in 5.30.
      state $left_brace = '\\{';
      state $covered = q/}()[]/;
      state $meta  = $left_brace . quotemeta( qq/${covered}@/ );

      state $punc = q/-=,._/;

      my $no_keep = qq/[^${punc}${alnum}${meta}]+/;
      $no_keep = qr/$no_keep/u;

      state $punc_or = join( q/|/,
                           $left_brace,
                           map { split '', quotemeta $_ } ( $covered, 
+$punc )
                        );

      state $many_seq = qr/[${punc}]{2,}/;
      state $pre_seq  = qr/[${punc}]+_/;
      state $post_seq = qr/_[${punc}]+/;

      my ( $base, $dir ) = fileparse( $old_path );
      for ( $base )
      {
         s/$no_keep/_/g;

         # Collapse same.
         s/($punc_or)\1/$1/g;

         # Collapse any sequence.
         s/$pre_seq/_/g;
         s/$post_seq/_/g;
         s/$many_seq/_/g;
      }
      return path_if_diff( $old_path, $dir, $base );
   }

   # Returns a cleaned path if possible; else C<undef>.
   #
   # It tries to keep a file path be a smaller set of characters for f
+iles on
   # Microsoft Windows.
   #
   # Nothing is replaced, only a warning is issued for file names that
+ match ...
   #
   #     CON, PRN, AUX, NUL, COM0, COM1, COM2, COM3, COM4, COM5, COM6,
+ COM7,
   #     COM8, COM9, LPT0, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, L
+PT8, and
   #     LPT9
   #
   # See https://learn.microsoft.com/en-us/windows/win32/fileio/naming
+-a-file that lists
   # ...
   #   Use any character in the current code page for a name, includin
+g Unicode
   #   characters and characters in the extended character set (128–25
+5), except
   #   for the following:
   #
   #    The following reserved characters:
   #        < (less than)
   #        > (greater than)
   #        : (colon)
   #        " (double quote)
   #        / (forward slash)
   #        \ (backslash)
   #        | (vertical bar or pipe)
   #        ? (question mark)
   #        * (asterisk)
   #
   #        Integer value zero, sometimes referred to as the ASCII NUL
   #        character.
   #
   #        Characters whose integer representations are in the range 
+from 1
   #        through 31, except for alternate data streams where these 
+characters
   #        are allowed
   #        ...
   #   Do not use the following reserved names for the name of a file:
   #
   #     CON, PRN, AUX, NUL, COM0, COM1, COM2, COM3, COM4, COM5, COM6,
+ COM7,
   #     COM8, COM9, LPT0, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, L
+PT8, and
   #     LPT9. Also avoid these names followed immediately by an exten
+sion; for
   #     example, NUL.txt and NUL.tar.gz are both equivalent to NUL. F
+or more
   #     information, see Namespaces.
   #
   #   Do not end a file or directory name with a space or a period. A
+lthough
   #   the underlying file system may support such names, the Windows 
+shell and
   #   user interface does not.  However, it is acceptable to specify 
+a period
   #   as the first character of a name. For example, ".temp".
   #   ...
   #
   sub cleanse_for_windows
   {
      my ( $old_path ) = @_;

      state $bad_char = q[<>:"|?*]
                     . '\\'
                     . join( q[], map { chr } 0..31 )
                     ;

      my %sub_replace = ( qr/[^\x00-\xff]+/ => q[^],
                              q/(?:[.]|[ ]+)$/ => q[_],
                              qq/[$bad_char]/ => q[-],
                           );

      my ( $base, $dir ) = fileparse( $old_path );

      $base = prefix_windows_reserved( $base );
      for ( $base )
      {
         for my $found ( keys %sub_replace )
         {
            my $repl = $sub_replace{ $found };
            s{$found}{$repl}g;
         }
      }
      return path_if_diff( $old_path, $dir, $base );
   }


   # Returns the base name prefixed with "_" if it matches a reserved 
+word.
   sub prefix_windows_reserved
   {
      my ( $base ) = @_;

      # Prefix with "_".
      state $prefix = q[_];
      state $reserved = join( q[|],
                           qw[ CON PRN AUX NUL
                              COM0 COM1 COM2 COM3 COM4 COM5 COM6 COM7 
+COM8 COM9
                              LPT0 LPT1 LPT2 LPT3 LPT4 LPT5 LPT6 LPT7 
+LPT8 LPT9
                           ]
                        );
      state $regex = qq/^( (?: $reserved )(?:[.].+)? )\$/;

      $base =~ s{$regex}{$prefix$1}xi;
      return $base;
   }

}

[Corion]

Ooh - also looking at PRN and CON, my module Text::CleanFragment doesn't do that! Tt uses Text::Unidecode to downgrade accented text (etc) to something ASCII, since I don't trust cross-filesystem consistency for non-ASCII :)

[parv]

Thanks much for linking Text::CleanFragment & Text::Unidecode. First one seems to do everything that I would do for <Windows file systems that are not NTFS>. It would be perfect to replace my code if degradation of Unicode could be optionally skipped on file systems where Unicode is non-issue.

I wanted to use icu or similar (or, the second module) to preserve some information; due to lack of time (motivation, really) never looked into that. The second one may solve that issue for me. Sweet👍🏽

Oh, there is Unicode::ICU💡 So should create an another version that first tries Unicode::ICU; failing that, Text::Unidecode. (ugh no, no, do not want to go further down this rabbit hole for cannot help but think of Tom C's response on Stack Overflow (on WayBack Machine aka web.archive.org))

[afoken]

Learned the hard way that NTFS would allow file names to be written to FAT32 even if some characters are outside of FAT32 specification.

NTFS won't write to FAT32. NTFS is a filesystem just like FAT32. And so the NTFS driver will only ever write to NTFS-formatted media, as will the (V)FAT driver only ever write to (V)FAT-formatted media.

FAT32 isn't the problem here. Windows' history is the problem explaining both the "forbidden" names (AUX, CON, PRN, NUL, ...) and the "forbidden" characters.

The first problem is due to someone during the early history of MS-DOS deciding that having a separate \DEV\ directory for Unix-style devices was not needed, and so the devices just existed in every directory, invisible and not in the filesystem, but still kind-of there. Early versions of MS-DOS still accepted the \DEV\ directory, even if it did not exist. This grew to bug-compatibility. In DOS batch files, you could use IF EXIST SOMENAME to check if a file named SOMENAME existed. But EXIST did not find directories, only files. To check if a directory existed, you had to use a bug, where EXIST finds all of the "forbidden" device names, because they kind-of exist in every existing directory. So, you use IF EXIST SOMENAME\NUL to check if a directory named SOMENAME exists.

The second problem is that ancient MS-DOS with its strictly limited filenames mostly did get the job done without proper quoting rules. Yes, it was not possible to pass <, >, and a few other characters as arguments to a program, but it was just a little bit annoying. "Long filenames" (VFAT) came and removed a lot of filename restrictions, quotes were needed, but nobody made a clear set of rules. It was a mess, and became worse and worse, as quoting and unquoting needs to be implemented by the applications and it is handled differently by different implementations. See Re^3: Having to manually escape quote character in args to "system"? for some ugly details.

If you do not need to interact with DOS/Windows, e.g. because your firmware expects a FAT filesystem for booting (UEFI-based PCs, Raspberry Pi, embedded systems, ...), you are free to use "forbidden" names and characters on a FAT filesystem. You could even make the filesystem case-sensitive or store long ("VFAT") filenames as UTF-16 instead of UCS-2. The firmware might expect some DOS compatibility, so you will probably not be able to boot your system from a file named "\" or "<". But perhaps it will boot from "AUX".

Alexander

[parv]

NTFS won't write to FAT32. NTFS is a filesystem just like FAT32. And so the NTFS driver will only ever write to NTFS-formatted media, as will the (V)FAT driver only ever write to (V)FAT-formatted media.

Perhaps that was bad|sloppy phrasing on my part. What do you call the operation of copying files from NTFS (Windows 10) to FAT32 (Windows knows of the removable mass storage device) then?

[afoken]

What do you call the operation of copying files from NTFS ... to FAT32 ... then?

Copying. Or, if you look at the filesystem driver level, reading from NTFS and writing to VFAT.

---

Regarding FAT names - see also File Allocation Table:

FAT-12, FAT-16, FAT-32 specify the size of a FAT entry, in bits, and thus indirectly also the maximum filesystem size.

VFAT extends the FAT by adding long filenames (everything beyond "8.3", including mixed case, spaces, and more allowed characters).

Linux has several closely related filesystem drivers, "fat" is a common part of the "msdos", "umsdos", and "vfat" drivers. The "msdos" driver is for plain old DOS disks, "8.3" names, no extras. "umsdos" (no longer enabled by default) implements Unix file attributes and long filenames on top of a plain, DOS-compatible FAT filesystem, by storing extra data in a special file per directory. "vfat" implements FAT with long filenames à la Microsoft.

ExFAT (as found on SDXC-Cards) and FATX (Xbox) do use a FAT, but with very incompatible structures due to the very different feature set.

Alexander