Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re^6: Getting started with MCE (the Many-Core Engine)

by Anonymous Monk
on Jun 13, 2018 at 15:30 UTC ( #1216567=note: print w/replies, xml ) Need Help??


in reply to Re^5: Getting started with MCE (the Many-Core Engine)
in thread Getting started with MCE (the Many-Core Engine)

Some helpful tips for processing a large array.

Greetings Mario. What a marvelous coincidence. I was just writing a question on both topics you address. First, thank you for the free software! MCE is amazing.

I've been tweaking it and noticed that smaller chunks are faster. For some reason 11k is the fastest here. I also noticed some delay in shutdown due to not calling shutdown but didn't know what was wrong. Thanks for the clue. Another issue I've come across is the value of MCE::Util::get_ncpu. On my machine (i7) it says 8 and of course it spawns 8 helpers but for some reason it's faster when I set that to 4:

> time mce volume: 11881376 chunk_size: 16000 max_workers: 8 real 0m19.776s user 0m43.350s sys 0m10.321s > time mce volume: 11881376 chunk_size: 16000 max_workers: 4 real 0m17.615s user 0m27.495s sys 0m4.836s

Replies are listed 'Best First'.
Re^7: Getting started with MCE (the Many-Core Engine)
by marioroy (Priest) on Jun 13, 2018 at 20:04 UTC

    Greetings,

    Depending on the processor, not all cores are "real" cores. Regarding chunk_size, I typically do not go over 8k for a large array and let chunking do its thing. Spawning workers early is beneficial, prior to allocating large amount of memory for input data.

    max_workers => MCE::Util::get_ncpu() / 2,

    Regards, Mario

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1216567]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (6)
As of 2019-04-18 20:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    I am most likely to install a new module from CPAN if:
















    Results (105 votes). Check out past polls.

    Notices?
    • (Sep 10, 2018 at 22:53 UTC) Welcome new users!