Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things

Re^3: randomising file order returned by File::Find

by BrowserUk (Pope)
on Mar 01, 2011 at 15:07 UTC ( #890768=note: print w/replies, xml ) Need Help??

in reply to Re^2: randomising file order returned by File::Find
in thread randomising file order returned by File::Find

pre-scanning at the beginning does not work in practice (for reasons including load-balancing due to different processing time (as mentioned) , the fact that the files that are available to process can change, that I often have to kill off jobs when other people want to use the cluster, etc, etc).

Another advantage of the file scanner server idea is that if you need to pause the 100+ processing clients, you only need instruct the server to stop responding to requests. Then those clients stop as soon as they've finished with their current file and just sit dormant waiting for a response.

When the cluster is free again, another single instruction to the server and they all kick off again, continuing their way down the list without any possibility of revisiting files already processed. Every file gets processed exactly once with none missed and no time wasted locking files and no possibility of race conditions.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
  • Comment on Re^3: randomising file order returned by File::Find

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://890768]
Discipulus Anonymous is fast and ironic
[Corion]: From time to time, you actually uncover things that always were wrong ;)
[Happy-the-monk]: Discipulus: please, what are the labels for CTRL+SHIFT+TAB on an Eatalian keyboards?
[Discipulus]: yes but ireally do not understand anonymous's reply about cpan r command: found no bug with that number and points to mjd.. doh?!

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (8)
As of 2017-11-21 09:56 GMT
Find Nodes?
    Voting Booth?
    In order to be able to say "I know Perl", you must have:

    Results (297 votes). Check out past polls.