Re: No Performance gain with Parallel::ForkManager

[walto]

Thanks for taking time to work on this.

McA suggested: Scan the whole directory tree and push all found mp3 files on a array (job queue), then split the queue into N parts (N = number of cores) and start N subprocesses working on their subqueue. In this case you do only fork once per subqueue and not for every file (which is expensive).

I can not find a simple way to split the loop into 2 (my no of cores) independent subprocesses. So I did what karlgoethebier suggested .
I tried with $MAX_PROCESSES=5 and $MAX_PROCESSES=2. The execution time is about the same and interestingly the unforked script does the job in about half the execution time.

[davido]

...and interestingly the unforked script does the job in about half the execution time.

...because the unforked process isn't causing your disk heads to seek back and forth repeatedly as each forked process grinds away at the same physical resource.

Sometimes forked processes work like a bucket brigade where the person filling the buckets does it very quickly, and hands them off to the workers who have to run from the water source to the fire and back before they request another full bucket. Other times, it works more like the person doing the filling is standing next to the fire, and has to keep running back and forth between the workers and the water source. You might be in the second situation.

[walto]

Yes, since the script is basically reading and writing mp3 tags it has mostly IO processes. And as you mentioned the bottleneck is the hard disk. It would be different if there are more computational processes.

[McA]

Hi,

probably something like that:

#!/usr/bin/perl
use strict;
use warnings;
use strict;
use 5.010;

use local::lib './lib';
use File::Find::Rule;
use Parallel::ForkManager;
use utf8;

my $dir = './';

my @mp3s = File::Find::Rule->file()
    ->name( '*.mp3' )
    ->in($dir);

say "INFO: Number of mp3 files found " . scalar @mp3s;

my $num_of_cores = 2;

my $pm = Parallel::ForkManager->new($num_of_cores);
for my $proc_id (0..$num_of_cores - 1) {
    my $pid = $pm->start and next;

    for(my $i = $proc_id; $i < scalar @mp3s; $i += $num_of_cores) {
        my $file = $mp3s[$i];
        say "Proc: $proc_id: Working on '$file'";
    }
    $pm->finish;
}
$pm->wait_all_children;
say "END";

I'm pretty sure you see the building blocks.

Regards
McA

[RichardK]

There's no need to be quite that verbose as Parallel::ForkManager handles the number of processes for you. So you can write that as :-

...
my $pm = Parallel::ForkManager->new($num_of_cores);
for my mp3 (@mp3s) {
   $pm->start and next;
   do_work($mp3);
   $pm->finish;
}
$pm->wait_all_children;

[McA]

Hi Richard,

am I wrong or does your solution fork a child per file found?

I wanted to give walto a snippet where I show how to spawn as many subprocesses as cores are available and the subprocesses working on a subqueue.

The verbose code tries to show that the subprocesses iterate over the initially created array in a way that they can "share" (*) this array without doing work twice. Have I overseen something?

UPDATE: It was an answer to the question "I can not find a simple way to split the loop into 2 (my no of cores) independent subprocesses...".

Best regards
McA

(*) It's a copy in the subprocess.

[walto]

Thanks for your example!

Adding an extra loop for the no of cores brought an speed improvement of the parallel execution of 100%.

Time parallel execution
real 0m0.312s, user 0m0.304s, sys 0m0.052s
Time serial exection
real 0m0.324s, user 0m0.272s, sys 0m0.031s

That's approx 4% overall gain.