Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer

Re^3: Function to sweep a file tree

by vr (Curate)
on Jun 21, 2020 at 11:42 UTC ( #11118309=note: print w/replies, xml ) Need Help??

in reply to Re^2: Function to sweep a file tree
in thread Function to sweep a file tree

Hi, glad to hear your script was improved. To your questions:

(1) Re: S_IFDIR. Easy to check: as I see there are 2 entries/directories in my C:/Users tree with FILE_ATTRIBUTE_REPARSE_POINT bit set; for both S_IFDIR is also set. So, to strictly follow Win32::LongPath documentation as to what to treat as "directory" i.e. exclude such entries, -- use fragment you quoted from documentation. Stress tests are good, but my impression was your tree to monitor should be known beforehand, whether it can or cannot contain reparse points. And, I meant the same thing (about "controlled environment"), when I said built-in readdir might be enough vs. readdirL, if a tree is known to be grown by native speakers/users of single (system) Windows code page. For versatility, sure, Win32::LongPath should be preferred.

(2) Re: <C:/Users/bo/Application Data/> - I don't understand (was this a question/problem?)

About freezes: that's not good. I can repeatedly sweep C:/Users and C:/Windows without issues, both with my and your scripts. At first, I suspected iterative readdirL (furthermore interspersed with lstatL calls) might be the reason, but, no, your script runs OK here. Still, to debug, maybe try to switch from iterative use to list context call. BTW, I don't observe any noticeable speed difference.

Lastly, about different total results of (various) Perl techniques and/or what Explorer reports: if you really want to pursue to the core, there's dichotomy with extensive logging, but of course you know the method.

Replies are listed 'Best First'.
Re^4: Function to sweep a file tree
by bojinlund (Prior) on Jun 22, 2020 at 05:56 UTC

    About freezes:

    I have made a script with
    foreach my $done ( 1 .. 100 ) { FS_sweep( 'C:', \&FS_file_big ); ...

    Running this script several times in Emacs, using the compile command, all resulted in a freeze. The the number of calls to FS_sweep before freeze was between 0 and 91.

    Running the script in the command shell resulted in 0 to 14 succeeded calls to FS_sweep. In most cases none.

    <C:/Users/bo/Application Data/> (was this a question/problem?)

    use strict; use warnings; use 5.010; use Data::Dump qw(dump dd ddx); use Win32::LongPath; use Fcntl ':mode'; sub do_readdirL_arr { my $dir_path = shift; my $dir = Win32::LongPath->new; unless ( $dir->opendirL($dir_path) ) { warn "!! unable to open $dir_path ($^E)"; return; } my @name = $dir->readdirL(); return \@name; } sub do_readdirL_while { my $dir_path = shift; my $dir = Win32::LongPath->new; unless ( $dir->opendirL($dir_path) ) { warn "!! unable to open $dir_path ($^E)"; return; } my @name; while ( my $name = $dir->readdirL() ) { push @name, $name; } return \@name; } my @path = ( 'C:/Documents and Settings', 'C:/ProgramData/Application +Data', 'C:/ProgramData/Desktop', 'C:/ProgramData/Start Menu', 'C:/ProgramData/Templates', #'C:/Users', 'C:/Windows/appcompat/Programs', 'C:/Windows/System32/Com/dmp +', #'C:/Windows/System32/spool', 'C:/Windows/System32/Tasks', 'C:/Windows/System32/Tasks_Migrated', 'C:/Windows/SysWOW64/Com/dmp +', 'C:/Windows/SysWOW64/Tasks', 'C:/Windows/Temp', ); foreach my $path (@path) { say "$path: array: ", dump( do_readdirL_arr($path) ), ' while: ', dump do_readdirL_while($path); } __DATA__ C:/Documents and Settings: array: ["g"] while: ["~"] C:/ProgramData/Application Data: array: ["u"] while: ["{"] C:/ProgramData/Desktop: array: ["H"] while: ["|"] C:/ProgramData/Start Menu: array: ["~"] while: ["m"] C:/ProgramData/Templates: array: ["K"] while: ["e"] C:/Windows/appcompat/Programs: array: ["u"] while: ["Q"] C:/Windows/System32/Com/dmp: array: ["m"] while: ["l"] C:/Windows/System32/Tasks: array: ["\\"] while: ["\x7F"] C:/Windows/System32/Tasks_Migrated: array: ["\22"] while: ["\f"] C:/Windows/SysWOW64/Com/dmp: array: ["p"] while: ["u"] C:/Windows/SysWOW64/Tasks: array: ["["] while: ["p"] C:/Windows/Temp: array: ["\x{105}"] while: ["\x{110}"]

    __DATA__ contains an example of output. It changes for every run.

    I have no access to the directories using the MS File explorer.

    When I in FS_sweep skip those directories, 'C:/Users' and C:/Windows/System32/spool' there are no freezes.

      Now it's getting sad/worrisome/interesting. With my old 5.26 Perl, your script above reports empty arrays as expected (and, as a side note, some reasonable content if run with administrative rights). Also, I finally did mega-stress-test (sweep C:/ for 100 times), it completes OK.

      Then I tried latest recommended Strawberry Perl versions,,, -- under 5.28 and 5.30 results are as you report: random bytes/single characters for "protected operating system" (as MS UI calls them) folders. The "mega-test" (tried once) hung on 5th sweep, though I was able to ctrl-C it. So, definitely looks like a bug (somewhere in Win32::LongPath XS guts?).

      Update. Actually, the bug is not related to "special system" directories, and I'm no longer sure Win32::LongPath under 5.26 behaved correctly, too. Maybe the call to opendirL shouldn't succeed in the first place.

      >perl -v This is perl 5, version 30, subversion 2 (v5.30.2) built for MSWin32-x +64-multi-thread Copyright 1987-2020, Larry Wall Perl may be copied only under the terms of either the Artistic License + or the GNU General Public License, which may be found in the Perl 5 source ki +t. Complete documentation for Perl, including FAQ lists, should be found +on this system using "man perl" or "perldoc perl". If you have access to + the Internet, point your browser at, the Perl Home Pa +ge. >mkdir abc >perl -MData::Dump -MWin32::LongPath -E "for (1..5) {my $lp=Win32::Lon +gPath->new; $lp->opendirL('abc') or die; dd [$lp->readdirL]}" [".", ".."] [".", ".."] [".", ".."] [".", ".."] [".", ".."] >icacls abc /deny Everyone:F processed file: abc Successfully processed 1 files; Failed processing 0 files >perl -MData::Dump -MWin32::LongPath -E "for (1..5) {my $lp=Win32::Lon +gPath->new; $lp->opendirL('abc') or die; dd [$lp->readdirL]}" ["\b"] ["\b"] ["\35"] ["\32"] ["'"] >perl -MData::Dump -MWin32::LongPath -E "for (1..5) {my $lp=Win32::Lon +gPath->new; $lp->opendirL('abc') or die; dd [$lp->readdirL]}" ["\3"] ["\26"] ["#"] ["\22"] [4] >perl -MData::Dump -MWin32::LongPath -E "for (1..5) {my $lp=Win32::Lon +gPath->new; $lp->opendirL('abc') or die; dd [$lp->readdirL]}" ["M"] ["~"] ["M"] ["w"] ["I"] >perl -MData::Dump -MWin32::LongPath -E "for (1..5) {my $lp=Win32::Lon +gPath->new; $lp->opendirL('abc') or die; dd [$lp->readdirL]}" ["M"] ["}"] ["C"] ["l"] ["}"] >icacls abc /remove:d Everyone processed file: abc Successfully processed 1 files; Failed processing 0 files >perl -MData::Dump -MWin32::LongPath -E "for (1..5) {my $lp=Win32::Lon +gPath->new; $lp->opendirL('abc') or die; dd [$lp->readdirL]}" [".", ".."] [".", ".."] [".", ".."] [".", ".."] [".", ".."] >rmdir abc >

        Thanks vr for the help!

        I am using Strawberry Perl (64-bit)

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11118309]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (3)
As of 2021-09-26 14:11 GMT
Find Nodes?
    Voting Booth?

    No recent polls found