Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked

Re^3: Optimizing with Caching vs. Parallelizing (MCE::Map) (Better caching?)

by vr (Curate)
on Apr 08, 2020 at 21:40 UTC ( #11115238=note: print w/replies, xml ) Need Help??

in reply to Re^2: Optimizing with Caching vs. Parallelizing (MCE::Map) (Better caching?)
in thread Optimizing with Caching vs. Parallelizing (MCE::Map)

Hi marioroy,

I think I'm having an issue with MCE on Windows here, timing your code (collatz_vr):

use Time::HiRes 'time'; my $t = time; # the whole script here say time - $t; MCE::Flow->finish; say time - $t; __END__ 1 worker: 6.01482510566711 7.76495599746704 2 workers: 4.12953305244446 7.07751798629761 4 workers: 3.33010196685791 8.4802930355072

1st measurement approximately matches your output, but 1.5 - 2 seconds per worker to shutdown doesn't look OK to me.

For vr's demo, every worker starts with an empty cache. Meaning that workers do not have cached results from prior chunks. This is the reason not scaling as well versus the non-cache demonstrations

In other words, lots and lots of work is needlessly duplicated.

_cache_collatz( $_ ) for 1 .. 1e6; say scalar %cache; %cache = (); _cache_collatz( $_ ) for 1 + 4e5 .. 1e6; say scalar %cache; __END__ 2168611 2168611

So, effectively, in situation with e.g. 10 workers, 4 junior workers, filling cache for ranges up to 400_000, are free to slack off and not gather their results at all, and there's large amount of overlap in work of 6 senior workers, too. For now, I have no solid idea how to parallelize this algorithm efficiently.

Replies are listed 'Best First'.
Re^4: Optimizing with Caching vs. Parallelizing (MCE::Map) (Better caching?)
by marioroy (Parson) on Apr 09, 2020 at 08:30 UTC

    Hi, vr

    The 1.5 - 2 seconds is likely threads freeing memory. Change my %cache to our %cache. MCE defaults to spawning threads on Windows. Try passing the MCE option use_threads => 0 to ensure emulated child instead.

    our %cache = ( 1 => [ 1, 1, undef ]); # for faster global cleanup mce_flow_s { max_workers => MCE::Util::get_ncpu(), gather => \@sizes, bounds_only => 1, use_threads => 0, }, sub { }, 1, 1e6;

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11115238]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (4)
As of 2021-09-18 17:45 GMT
Find Nodes?
    Voting Booth?

    No recent polls found