Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: Splitting up a filesystem into 'bite sized' chunks

by zork42 (Monk)
on Jul 13, 2013 at 11:13 UTC ( #1044143=note: print w/replies, xml ) Need Help??


in reply to Splitting up a filesystem into 'bite sized' chunks

Just a 'find' takes a substantial amount of time on this filesystem (days).
  1. Any idea why it takes so long?
  2. Does find or File::Find spend most of the time waiting for a response from the remote server?
  3. Would you be able to speed up File::Find by searching multiple directories simultaneously?
    If you go down a few directory levels and find (say) 100 subfolders, could you search each of those subfolders simultaneously?
  4. Would you be able to speed up File::Find by using RPCs?
  • Comment on Re: Splitting up a filesystem into 'bite sized' chunks

Replies are listed 'Best First'.
Re^2: Splitting up a filesystem into 'bite sized' chunks
by Preceptor (Deacon) on Jul 16, 2013 at 18:52 UTC

    Contention and sheer number of files, mostly. Parallel traversals will help if I divide the workload reasonably - I've got a lot of spindles and controllers. Some filesystems will work ok with a 'traverse down' approach, but others are much more random in distribution. I don't want to extend my batches too much, because of outages, glitches etc.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1044143]
help
Chatterbox?
[Discipulus]: mah, winbuntu 10...
[marto]: whent he boys start school I'll build myself as new machine, ryzen based
[Corion]: marto: That's cool! I think my (home) desktop spends 30 seconds in the BIOS...
[Corion]: marto: Heh - currently they seem promising, but I think I'll stay with nVidia+Intel for the time being, as I've been bitten too often by bad AMD drivers
Discipulus is (still?) not a big fan of notepad++
[marto]: Corion, I think in the past this was a big problem for them. GPU driver wise the improve all the time. I use the open source drivers on my machine (old R9 270, 2GB) and had no problems
[Discipulus]: is not the problem with new hardware?
[marto]: notepad++ ticks all the boxes for me, it's a fine line between just enough, and bloat, in editors for me :P
[marto]: Discipulus]: less so than the modern intel CPUs
[marto]: also, AMD are faster at fixing the CPU microcode it would seem

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (8)
As of 2017-07-27 08:58 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    I came, I saw, I ...
























    Results (407 votes). Check out past polls.