Re^2: Fastest way to recurse through VERY LARGE directory tree

[runrig]

Unless the OP needs the entire list of "tens of millions of files", I'd suggest not using File::Find::Rule, and I'd instead suggest an iterator or callback based routine (like File::Find). If you can process files one at a time, there's no need to build such a huge list.

[eff_i_g]

runrig,

I don't follow you. File::Find::Rule does not simply return every file (although it can). You can instruct it what to return based on type, size, name, and even make a determination via a custom sub. You can have the sub perform actions and ignore the larger return value, or iterate with the start and match methods.

[runrig]

Read the OP. I didn't say FFR has to return the entire list, the OP says the entire list needs to be processed. So, in this case, FFR would return the entire list of files.

[eff_i_g]

Ah, gotcha. But you can work around building the entire list:

use warnings;
use strict;
use File::Find::Rule;

File::Find::Rule->file->exec(
    sub {
        print $_[2], "\n";
        return 0;
    }
)->in('/path');