Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re: Parallel processing with ForkManager

by zentara (Archbishop)
on Sep 20, 2012 at 11:27 UTC ( #994636=note: print w/ replies, xml ) Need Help??


in reply to Parallel processing with ForkManager

I haven't been able to find good examples of using ForkManager to read a file a line-at-a-time and then doing something with those lines.

This isn't exactly your scenario, but it may help get you going.

#!/usr/bin/perl use warnings; use strict; use Parallel::ForkManager; my $dir = shift || '.'; my @dirs = get_sub_dirs($dir); my $max_tasks = 3; my $pm = new Parallel::ForkManager($max_tasks); $|++; my $start = time(); for my $dir (@dirs) { my $pid = $pm->start and next; printf "Begin processing $dir at %d secs.....\n", time() - $start; #push all the $dir/files into @ARGV and search through them #line by line @ARGV = <$dir/*>; while (<ARGV>) { close ARGV if eof; #find some search term, using "perl" for example if( $_ =~ /perl/) { print "$ARGV: $. :$_\n"; $pm->finish; goto END; } } END: printf ".... $dir done at %d secs!\n", time() - $start; $pm->finish; } print " all done\n"; exit; ########################################################## sub get_sub_dirs { my $dir = shift; opendir my $dh, $dir or die "Error: $!"; my @dirs = grep { -d $_ } readdir $dh; @dirs = grep !/^\.\.?$/, @dirs; closedir $dh; return @dirs; }

I'm not really a human, but I play one on earth.
Old Perl Programmer Haiku ................... flash japh


Comment on Re: Parallel processing with ForkManager
Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://994636]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (7)
As of 2015-07-08 02:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (93 votes), past polls