Re: Splitting up a filesystem into 'bite sized' chunks

[zork42]

Just a 'find' takes a substantial amount of time on this filesystem (days).

1. Any idea why it takes so long?
2. Does find or File::Find spend most of the time waiting for a response from the remote server?
3. Would you be able to speed up File::Find by searching multiple directories simultaneously?
   If you go down a few directory levels and find (say) 100 subfolders, could you search each of those subfolders simultaneously?
4. Would you be able to speed up File::Find by using RPCs?

[Preceptor]

Contention and sheer number of files, mostly. Parallel traversals will help if I divide the workload reasonably - I've got a lot of spindles and controllers. Some filesystems will work ok with a 'traverse down' approach, but others are much more random in distribution. I don't want to extend my batches too much, because of outages, glitches etc.