Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

Re: File::Find redux: how to limit hits?

by graff (Chancellor)
on Jun 04, 2002 at 07:31 UTC ( #171422=note: print w/replies, xml ) Need Help??

in reply to File::Find redux: how to limit hits?

Well, I don't know if anyone else has noticed this (maybe my mileage is way different) but when I tried timing a simple File::Find approach (as provided by "find2perl") against the equivalent back-tick `find ...` command on my linux box, I got a 5-to-1 wallclock ratio:
use strict; use Benchmark; use File::Find (); # this part is unnecessary, but find2perl made it up, so I copied it: use vars qw/*name *dir *prune/; *name = *File::Find::name; *dir = *File::Find::dir; *prune = *File::Find::prune; timethese( 10, { 'File::Find' => \&find2perl, 'Shell:Find' => \&shellfind, }); my @found; # I'm not using this for anything at present sub find2perl { @found = (); File::Find::find({wanted => \&wanted}, '.'); # made by find2perl } sub wanted { # made by find2perl my ($dev,$ino,$mode,$nlink,$uid,$gid); (($dev,$ino,$mode,$nlink,$uid,$gid) = lstat($_)) && -d _ && push @found, $_; } sub shellfind { @found = `find . -type d`; } __END__ # OUTPUT: Benchmark: timing 10 iterations of File::Find, Shell:Find... File::Find: 27 wallclock secs (20.87 usr 4.01 sys + 0.01 cusr 0.00 +csys = 24.89 CPU) @ 0.40/s (n=10) Shell:Find: 5 wallclock secs ( 0.21 usr 0.01 sys + 1.28 cusr 3.36 +csys = 4.86 CPU) @ 45.45/s (n=10) # I printed scalar(@found) in one test, and these results # were obtained where there were over 6K directories under "."

So, maybe the first thing to try for speeding things up is:

# don't use File::Find;
Apart from that, have you considered an alternative like this:
  • run a separate process to create and maintain a set of file name listings for each path; the first time you run this, it'll take a long time, but thereafter, it only needs to find the directories that were created/modified since the last run, then for just those paths, diff the current file inventory against the previous file name list, and write the set of new files to a separate log file. (one approach is given below)
  • adapt your re-indexing job so that it works from the log of new files, and it doesn't use find at all.

Of course, you can put the two steps together into a single script.
#!/usr/bin/perl # Program: find-new-files.perl # Purpose: initialize and maintain a record of files in a # directory tree # Written by: dave graff # If a file called "paths.logged" does not exist in the cwd, we create # one, and treat all contents under cwd as "new". If "paths.logged" # already exists, we find directories with modification dates more # recent than this file, and treat only these as "new". # For each "new" directory, assume a file.manifest is there (create an # empty one if there isn't one), and diff that file against the curren +t # inventory of data files, storing all new files to an array. # Of course, this will fail in all paths where the current user does # not have write permission, but such paths can be avoided by adding # a suitable condition to the first "find" command. use strict; my $path_log = "paths.logged"; my ($list_name,$new_list) = ("file.manifest","new.manifest"); my $new_flag = ( -e $path_log ) ? "-newer $path_log" : ""; my @new_dirs = `find . -type d $new_flag`; # add "-user uname" and/or "-group gname" to avoid directories where # the current user might not have write permission my $diff_cmd = "cd 'THISPATH' && touch $list_name && ". "find . -type f -maxdepth 1 | tee $new_list | diff - $list_name | +grep '<'"; # the shell functions in $diff_cmd will: # - chdir to a given path, # - create file.manifest there if it does not yet exist, # - find data files in that path (not subdirs, not files in subdirs), # - create a "new.manifest" file containing this current file list, # - diff the new list of files against the existing file.manifest, # - return only current files not found in the existing manifest. # since it's a sub-shell, the chdir is forgotten when the sub-shell is + done. open( OUT, ">new-file-path.list" ); foreach my $path ( @new_dirs ) { chomp $path; my $cmd = $diff_cmd; $cmd =~ s{THISPATH}{$path}g; # the output of the shell command needs to be conditioned to have the # path string prepended to each file name (we can leave the new-line # in place at the end of the name): print OUT join "", map { s{^< \.(.*)}{$path$1}; $_ } ( `$cmd` ); # replace the old manifest: rename "$path/$new_list", "$path/$list_name" or warn "failed to update $path/$list_name\n"; } close OUT; `touch $path_log`;

update: fixed some commentary in the second program; realized that <br> is required after </ul>.

Replies are listed 'Best First'.
Re: Re: File::Find redux: how to limit hits?
by 914 (Pilgrim) on Jun 04, 2002 at 16:48 UTC
    Wow! graff, that's some help, if you just 'whipped up' that script in response to my problem...

    Happily, it'll be here at PM for everyone else to find too... i'm not sure it's applicable in this exact problem (though it'll be useful to me in another area... thanks! i hadn't even asked that question yet!), as i need the code to NOT depend on the local find command. I'd really prefer this script to be portable across Solaris and Linux systems, with differing find commands.

    i'll be "flattering" your timing script too, optomizing this thing with whichever technique seems best.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://171422]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (4)
As of 2017-01-19 06:03 GMT
Find Nodes?
    Voting Booth?
    Do you watch meteor showers?

    Results (168 votes). Check out past polls.