Shorten windows paths too long

[gj2666]

I tried super search before I decided to post here. I understand that Windows does not support pathnames longer than 256ish. However, I have paths longer than that on disk. It seems that on my Hudson build server (win32) Java/Maven/Hudson or some other evil entity insists on writing those "paths too foul" to my disk.

The evil Java build automation developers I work with want to move the Hudson jobs to a different location, however the paths are too long and windows just whines when I try to move them. (see output at bottom)

I could alter them manually by changing several paths manually, but whenever the job runs again it updates whatever was there with the current path too foul. Can't get them all renamed manually before new ones are created.

So I tried something I found here the other day, but it fails to execute the move command here: (And yes, I have wicked skills, but not in a good way.)

H:; cd $olddirname; cd ..; move /Y $olddirname $newname` or die "Can't
+ move $olddirname $newname: $!";

$olddirname is the full path to the dir I want to rename (shorten). $newname is the new name of the dir (no path) executed (via the ..) to position me in the parent dir of the one I want to change. My question is...what is another approach that I could make work since making a system call and using the dos commands seems to be a bust.

#!perl
#
# fixlength.pl
# Script to browse a file path (for hudson jobs)and find "target"
# directories and shorten the pathname by renaming several common
# directory names  dir /s /b Zz*
#    $File::Find::dir  = /some/path/
#    $_                = foo.ext
#    $File::Find::name = /some/path/foo.ext
# 
use strict;
use warnings;
use File::Find; #used to traverse a directory tree

my $count;
my $char;
my @pathname;
# open (FILE, "<$file") or die "Can't open $file: $!";
my $directory = "H:/softwaredistribution/hudson/jobs/";

finddepth(\&process_file, $directory);
#----------------------------------------------------------------
# Sub Routines start here
#----------------------------------------------------------------   
sub process_file{

    foreach (my $dir = $File::Find::dir){
        my $dir1=$File::Find::name;
        if (($_ ne "\.")&&($_ ne "\..")){
            print "Dir is : $File::Find::dir \n file is  :$_ \n and fu
+llpathname is : $File::Find::name \n";
            my %rephash=("target"=>"t", "checkout"=>"co", "was6-maven-
+plugin"=>"w6","SNAPSHOT"=>"S");
            my @keys = %rephash;
            my @value = %rephash;
            
            foreach my $key (sort keys %rephash){
                if ($dir1=~m/$key$/) {
                    my $olddirname=$dir1;             
                    $_=$dir1;
                  #  s/$key/$rephash{$key}/g;
                    my $newname=$rephash{$key};
                    print "Command is H:; cd $olddirname; cd ..; move 
+/Y $olddirname $newname \n";
                    `H:; cd $olddirname; cd ..; move /Y $olddirname $n
+ewname` or die "Can't move $olddirname $newname: $!";
                }
            } 
        }                
    }
}

Output:
H:/softwaredistribution/hudson/jobs/Conservation_Payment_Release/workspace/trunk/target/checkout/conservation-ejb-batch/target/was6-maven-plugin; cd ..; move /Y H:/softwaredistribution/hudson/jobs/Conservation_Payment_Release/workspace/trunk/target/checkout/conservation-ejb-batch/target/was6-maven-plugin w6

Can't move H:/softwaredistribution/hudson/jobs/Conservation_Payment_Release/workspace/trunk/target/checkout/conservation-ejb-batch/target/was6-maven-plugin w6: at O:\RatlSupport_ML_dev\SRM-SoftwareDevCV\RatlSupport\Scripts\fixlength.pl line 42.

Line 42 is the `H:;...` In the output above we're trying to move (rename was6-maven-plugin to "w6").

All the output in trunk and below gets rewritten on build machine each new build and never commits to the SVN repository to it's okay to change on this server.

Thank you for your time and patience. gj

[tye]

Can't you just trim the huge path and give 'move' the key not the full path so that you try to run this instead:

cd /d H:/softwaredistribution/hudson/jobs/Conservation_Payment_Release
+/workspace/trunk/target/checkout/conservation-ejb-batch/target; move 
+/Y was6-maven-plugin w6

Update: You don't appear to be that close to the 255 point when the failure happens so the problem is probably that the directory is 'in use' in a way that prevents it from being renamed. The move fails but produces no error message? That is strange. You could get the old SysInternals filemon and see the error reason for the move failing or use Perl's rename and report $! and $^E after that fails.

[gj2666]

tye, I'll try cd'ing to the dir and passing in just the pathnames.

Yes, it is really and truly too long. I'm trying to recover some length by shortening some of the directories near the start of the path in order to be able to move them without doing them one at a time by hand.

H:\so123456789012345ion/hudson/jobs/BFffs/workspace/b1234567-file-pare
+nt/target/checkout/persistence-dom/src/main/java/xyz/abcd/fqq/perdom/
+pc/farmoperatingplancontributionpersistentservice/.svn/text-base/Stor
+eManagementContributionDeclarationPC.java.svn-base,

258 characters

But you're right, the error occurs on a node in the path way before length becomes a problem. It works manually and that's what has me stumped.

I am getting "Bad File Descriptor" in the files that I don't want to change anyway, but on the ones that I want to change I get nothing. I'm debugging in komodo interactively so if there were anything there I should see it on $! as I step through the code.

[Marshall]

Ok, now I see that indeed you have a 258 char path. Once you exceed one of these "magic power of 2" boundaries, the results can be unpredictable, often due to due to lower level things related to memory allocation. In this case, I suspect that you have a classic buffer overflow problem.

Sometimes you "get away with it" and sometimes not! The command line version may use different memory allocation than some other version - this is in any event, "fragile" code that depends upon being lucky!

It appears to me that (a) you have a relatively small number of paths (70), that need to be shorted by a very modest amount - at least 3 chars, but I would try to go more than that for later expansion.

You could build a translation table. Say shortening "persistence-dom" to just "persistD". That alone would solve your immediate problem and get total path characters to less than 255. Throw some other translations in there and you can get to less than 200 chars.

There is a post in this node about "subst" which is one way to implement the "mount point" idea that I suggested earlier, and that is one way to go.

There is also a post that suggests an obvious solution: Don't generate path's with more than 255 chars! Explain to your users why this is important and a big pain in the ass to deal with and they will understand it.

The best solution is that everybody generates stuff that is easy to deal with, understands what the rules are, and why that makes sense. I would counsel you to get out of the file path/name translation business. If you don't, this will be a never ending assignment for you. In short, talk to your users, explain what grief this causes and some simple steps that they can do which eliminates this grief and is better for everybody.

[gj2666]

Apparently not. I will continue my quest...

[Marshall]

The limit on paths with NTFS is 255 chars that is a hard limit and there is no tricky OS setting to change that of which I am aware. One thing that you could try is making a mount point further down the directory tree. That should allow you to get to a file with huge absolute pathname using a shorter pathname which uses this "fake" drive letter. NTFS mount points As they say, "some assembly" required... There are only 26 drive letters possible: A-Z so this is not nearly as flexible as a unix mount point.

I suppose that you could make some sort of "movelong" command that copies the file twice. CD down into the tree to target, then copy file from current directory to some path like C:/temp, then traverse tree to reach new directory and copy from C:/temp to current directory?

Update: NTFS maintains a "short" filename for backward compatibility (unless that feature is turned off on your system). how Windows generates short filenames. To see what these are do a "dir /x". There should be some way to assemble enough of these short names together to create a path that fits within 255 chars with target file being a "long name".

I hadn't counted characters in your path. I just assumed that you were right about the 255, but your path does indeed appear to be short enough as per tye's post. If any file in the path is open, even for read, windows won't let you rename the path, but you should get an error message.

[afoken]

There is a way to work around with paths longer than 260 chars: http://www.foken.de/alexander/projekte/windows/deepcopy.html#Algorithm.

Alexander

[gj2666]

Thank you for the suggestion, it looks like something that would do the trick. I will likely have to give blood before they'll allow me to download it unfortunately.

Governments are fussy that way.

[tye]

`H:; cd $olddirname; cd ..; move /Y $olddirname $newname` or die "Can'
+t move $olddirname $newname: $!";

That is pretty broken code. `` (qx<>) doesn't return a boolean. It returns the output of the commands. So you are dieing if that command produces no output. And $! usually is irrelevant after qx() fails.

my $output= qx(H:; cd $olddirname; cd ..; move /Y $olddirname $newname
+);
if(  0 != $?  ) {
    die "Can't move $olddirname $newname: $output";
}

Might be closer to correct.

[gj2666]

I stand corrected *blushing* I admit my code is remedial at best. I wasn't sure how to do that and gave my best guess. Thank you for the correction there. I'll fix my version per your recommendations.

Thanks again,

[tokpela]

You should use the DOS subst command which you can use to shorten your path.

The Win32::FileOp module has a Subst routine if you want to do this from within your script if your path is too long.

[Anonymous Monk]

Maybe you should petition to rename a few directories to make it a little more succinct.

[gj2666]

That is exactly what I'm trying to do here. Rename directories programmatically to make the paths shorter i.e. more succinct. I have something like 70 of them and each path requires about 4 or 5 renames to shorten by 30 or so characters.

Trying to get them all done in one day to move the whole mess to a shorter path root. (i.e. c:\hw instead of c:\softwaredistribution

Unfortunately we can't move the folders until the path is < 256 characters.

[SuicideJunkie]

What if you were to build a multilevel hash to lookup the conversions?

BFS the filesystem tree, and rename the first folder you see to 'a', the second to 'b' and so on using the magic increment. Recurse down the tree until you have 'c:\RootOfEvil\a\b\a\c\a\a\f\a\b\filename.ext'

Your hash tree would tell you that $treeHash->{a}{b}{a}{c}...{f}{a}{b} was originally named 'debug' and $treeHash->{a}{b}{a}{c}...{f}{a} was originally named 'bin', and so on, so that you can undo the renames later.

You'd end up renaming 'c:\RootOfEvil\BigNameLevel1' to 'c:\RootOfEvil\a', and then moving on to rename 'c:\RootOfEvil\a\BigNameLevel2' to 'c:\RootOfEvil\a\a' and so on.




Or, much more simply... what if you just pile the whole tree into an archive, copy that across, and then unpack it on the other side, leaving the archiver app to deal with the excessive tree :)

[wwe]

I'm not sure if perl and all of the modules you are using are aware of this but try the solution from here http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx There is another API which is not limited to 260 characters but 32768. This Unicode API can be addressed by using a '\\?\' prefix e.g. '\\?\C:\temp\' or for network shares '\\?\UNC\server\c$\temp'. See attached link for details.

Update: corrected wrong link.

[gj2666]

Hmmm, very interesting. Thanks for the tip.