Re: Re: Efficient processing of large directory

[BrowserUk]

It's worth noting that if your trying to find a subset of the files contained in a subdir, rather than processing them all, then using <*.txt> is considerably faster that using either File::Find or opendir/readdir/closedir. At least that is the case under Win32 as the wildcard matching is done by the OS and only those files matching are past back.

In the examples below, the first comparison shows selecting all 17576 files in a subdirectory. In this case, glob and File::Find come out pretty much even.

In the second comparison, a subset of 676 files is selected from the 17000 using a wildcard. In this case, the glob runs 650% faster as it is only processing the 676, rather than looping over the whole 17000+.

Of course, if any real processing was being done rather than just counting the files, the difference would rapidly disappear.

In this case, the OP's use of the word "efficient" was most likely to do with the memory used by slurping all 17000 names in to memory rather than speed, but if not all those 17000 file are .txt files, the time saved might be worth having.

#! perl -slw
use strict;
use Benchmark qw[ cmpthese ];
use File::Find;

our( $dir, $glob, $re_glob );

my %tests = (
    GLOB => q[ my $count; $count++ while <${dir}/${glob}>;            
+   print 'GLOB: ', $count; ],
    FIND => q[ my $count; find( sub{ m[$re_glob] and $count++ }, $dir 
+); print 'FIND: ', $count; ],
);


( $dir, $glob, $re_glob ) = ( 'bigdir', '/*.txt', qr[\.txt$] );

cmpthese( 3, \%tests );

( $dir, $glob, $re_glob ) = ( 'bigdir', '*A.txt', qr[A\.txt$] );

cmpthese( 10, \%tests );

__END__

P:\test>glob-ff
FIND: 17576
FIND: 17576
FIND: 17576
GLOB: 17576
GLOB: 17576
GLOB: 17576

     s/iter FIND GLOB
FIND   29.6   --  -1%
GLOB   29.4   1%   --

FIND: 676
FIND: 676
FIND: 676
FIND: 676
FIND: 676
FIND: 676
FIND: 676
FIND: 676
FIND: 676
FIND: 676
GLOB: 676
GLOB: 676
GLOB: 676
GLOB: 676
GLOB: 676
GLOB: 676
GLOB: 676
GLOB: 676
GLOB: 676
GLOB: 676

     s/iter FIND GLOB
FIND   29.6   -- -87%
GLOB   3.97 646%   --