Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Re: Fastest way to recurse through VERY LARGE directory tree

by eff_i_g (Curate)
on Jan 21, 2011 at 16:03 UTC ( #883562=note: print w/replies, xml ) Need Help??

in reply to Fastest way to recurse through VERY LARGE directory tree

I suggest File::Find::Rule over File::Find. I've used it recently for Finding Temporary Files and it goes through ~1TB in under an hour.
  • Comment on Re: Fastest way to recurse through VERY LARGE directory tree

Replies are listed 'Best First'.
Re^2: Fastest way to recurse through VERY LARGE directory tree
by runrig (Abbot) on Jan 21, 2011 at 16:38 UTC
    Unless the OP needs the entire list of "tens of millions of files", I'd suggest not using File::Find::Rule, and I'd instead suggest an iterator or callback based routine (like File::Find). If you can process files one at a time, there's no need to build such a huge list.

      I don't follow you. File::Find::Rule does not simply return every file (although it can). You can instruct it what to return based on type, size, name, and even make a determination via a custom sub. You can have the sub perform actions and ignore the larger return value, or iterate with the start and match methods.

        Read the OP. I didn't say FFR has to return the entire list, the OP says the entire list needs to be processed. So, in this case, FFR would return the entire list of files.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://883562]
marioroy At a large firm, had to awaken a security guy a few times so that nobody would catch him sleeping.
marioroy On a large dairy farm, almost went off the road with a 10 wheeler from exhaustion. On the farm, almost went off the silo with a big tractor. At the Fransiscan monastery, almost slipped off the dome while painting it.
[marioroy]: In Saudi Arabia caught fire but didn't burn. I think angels exist.
Lady_Aleena clean up after dinner finally.
[Discipulus]: uch! so we are twice lucky to have MCE..;=)
[marioroy]: Working a midnight graveyard shift, on the way home ran out of gas. just 2 minutes later somebody stops with a small tank of gas. It's wierd. There's always someone nearby for some reason.
[Lady_Aleena]: I was the one who stop[ed in the middle of a cold night to pick up a small family walking to a gas station. I got a few gallons of free gas for the round trip, and the grandmother said she'd pray for me.
[marioroy]: Folks do not know it, but MCE was ridiculously difficult. At Niagra Falls, heard a voice on enabling Grace while in the mist boat. That 10x performance is possible.
[marioroy]: Lady_Aleena Angels on earth are real people. That's awesome.

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (9)
As of 2017-05-29 08:24 GMT
Find Nodes?
    Voting Booth?