Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

Re^8: Splitting up a filesystem into 'bite sized' chunks

by Preceptor (Deacon)
on Jul 10, 2013 at 22:59 UTC ( #1043583=note: print w/replies, xml ) Need Help??

in reply to Re^7: Splitting up a filesystem into 'bite sized' chunks
in thread Splitting up a filesystem into 'bite sized' chunks

File::Find let's you specify "prune" which enables or disables traversal. If you know your last checkpoint is /mnt/myhome/stuff/junk you can pattern match your file path, and turn off traversal until you get a match. (you may have to roll up your checkpoint a little if the target has been deleted in the interim).

That'll - hopefully - give me a restartable find. Being able to "skip head" in future (and thus distribute processing) may require a first pass, and tracking multiple checkpoints.

Thinking about it, some sort of start/finish and some way of compensating for "drift". But one checkpoint every 100k files takes a huge list down to merely large. (but still doesn't help your first pass, unless you can take some wild guesses for initial checkpoints and do that same drift compensation.)

  • Comment on Re^8: Splitting up a filesystem into 'bite sized' chunks

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1043583]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (5)
As of 2016-10-21 01:17 GMT
Find Nodes?
    Voting Booth?
    How many different varieties (color, size, etc) of socks do you have in your sock drawer?

    Results (284 votes). Check out past polls.