|Welcome to the Monastery|
Re: MCE: Slow IPC between child and gather process in parentby marioroy (Vicar)
|on May 02, 2018 at 14:47 UTC||Need Help??|
How is $output defined and populated? It's not shown in the code. Being on a Mac, is the folder where the output file resides disabled from Spotlight indexing? E.g. Apple Menu -> System Preferences -> Spotlight -> Privacy Tab
MCE Relay may be helpful to factor out freeze/thaw operations for large data. Please find below two demonstrations for solely testing IPC. The Gather and Relay demonstrations take 11.3 and 12.1 seconds respectively for 1 million iterations. Both handle above 50k iterations per second.
Iterator & Gather demonstration
Iterator & Relay demonstration
Update: Changed max_workers from 'auto' to 3 above. Basically, try 3 ~ 5 workers as the OP's example involves lots of IO. The MCE module was written in a way to handle thousands of IPC requests per second. For such examples that do 10's of thousands IPC requests, limiting the number of workers allows the OS to allocate more CPU time for the manager process. Chunking is another way.