Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: No Performance gain with Parallel::ForkManager

by walto (Pilgrim)
on Feb 23, 2014 at 14:39 UTC ( #1075892=note: print w/replies, xml ) Need Help??


in reply to No Performance gain with Parallel::ForkManager

Thanks for taking time to work on this.

McA suggested: Scan the whole directory tree and push all found mp3 files on a array (job queue), then split the queue into N parts (N = number of cores) and start N subprocesses working on their subqueue. In this case you do only fork once per subqueue and not for every file (which is expensive).

I can not find a simple way to split the loop into 2 (my no of cores) independent subprocesses. So I did what karlgoethebier suggested .
I tried with $MAX_PROCESSES=5 and $MAX_PROCESSES=2. The execution time is about the same and interestingly the unforked script does the job in about half the execution time.
  • Comment on Re: No Performance gain with Parallel::ForkManager

Replies are listed 'Best First'.
Re^2: No Performance gain with Parallel::ForkManager
by davido (Cardinal) on Feb 23, 2014 at 19:02 UTC

    ...and interestingly the unforked script does the job in about half the execution time.

    ...because the unforked process isn't causing your disk heads to seek back and forth repeatedly as each forked process grinds away at the same physical resource.

    Sometimes forked processes work like a bucket brigade where the person filling the buckets does it very quickly, and hands them off to the workers who have to run from the water source to the fire and back before they request another full bucket. Other times, it works more like the person doing the filling is standing next to the fire, and has to keep running back and forth between the workers and the water source. You might be in the second situation.


    Dave

      Yes, since the script is basically reading and writing mp3 tags it has mostly IO processes. And as you mentioned the bottleneck is the hard disk. It would be different if there are more computational processes.
Re^2: No Performance gain with Parallel::ForkManager
by McA (Priest) on Feb 23, 2014 at 17:11 UTC

    Hi,

    probably something like that:

    #!/usr/bin/perl use strict; use warnings; use strict; use 5.010; use local::lib './lib'; use File::Find::Rule; use Parallel::ForkManager; use utf8; my $dir = './'; my @mp3s = File::Find::Rule->file() ->name( '*.mp3' ) ->in($dir); say "INFO: Number of mp3 files found " . scalar @mp3s; my $num_of_cores = 2; my $pm = Parallel::ForkManager->new($num_of_cores); for my $proc_id (0..$num_of_cores - 1) { my $pid = $pm->start and next; for(my $i = $proc_id; $i < scalar @mp3s; $i += $num_of_cores) { my $file = $mp3s[$i]; say "Proc: $proc_id: Working on '$file'"; } $pm->finish; } $pm->wait_all_children; say "END";

    I'm pretty sure you see the building blocks.

    Regards
    McA

      There's no need to be quite that verbose as Parallel::ForkManager handles the number of processes for you. So you can write that as :-

      ... my $pm = Parallel::ForkManager->new($num_of_cores); for my mp3 (@mp3s) { $pm->start and next; do_work($mp3); $pm->finish; } $pm->wait_all_children;

        Hi Richard,

        am I wrong or does your solution fork a child per file found?

        I wanted to give walto a snippet where I show how to spawn as many subprocesses as cores are available and the subprocesses working on a subqueue.

        The verbose code tries to show that the subprocesses iterate over the initially created array in a way that they can "share" (*) this array without doing work twice. Have I overseen something?

        UPDATE: It was an answer to the question "I can not find a simple way to split the loop into 2 (my no of cores) independent subprocesses...".

        Best regards
        McA

        (*) It's a copy in the subprocess.

      Thanks for your example!

      Adding an extra loop for the no of cores brought an speed improvement of the parallel execution of 100%.

      Time parallel execution
      real 0m0.312s, user 0m0.304s, sys 0m0.052s
      Time serial exection
      real 0m0.324s, user 0m0.272s, sys 0m0.031s

      That's approx 4% overall gain.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1075892]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (8)
As of 2021-01-23 21:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?