Parallel::ForkManager is certainly a good tool for managing a bunch of processes all under the control of a single "master" process which, in your case, would be the one that reads the 100MB file. However, you need to be careful.

Things to consider include:

• How many parallel clients can the database handle before it becomes a significant bottleneck?
• What is the overhead of forking - it's almost certainly too high to naively fork a new process for processing each line in the file.
• What do you need to do with the data retrieved from the db? While Parallel::ForkManager can return data from each forked process, it fakes this up by going via the disk. Will this turn into an I/O bottleneck?
• What is the overhead of connecting to the DB, and how can you reduce that?

This isn't exactly your scenario, but it may help get you going.

#!/usr/bin/perl
use warnings;
use strict;
use Parallel::ForkManager;

my $dir = shift || '.';
my @dirs =  get_sub_dirs($dir);

my $max_tasks   = 3;
my $pm = new Parallel::ForkManager($max_tasks);
$|++;
my $start = time();

for my $dir (@dirs) {
    my $pid = $pm->start and next;
    printf "Begin processing $dir at %d secs.....\n", time() - $start;
    
    #push all the $dir/files into @ARGV and search through them 
    #line by line
    
    @ARGV = <$dir/*>;
    
     while (<ARGV>) {
        close ARGV if eof;
       #find some search term, using "perl" for example  
       if( $_ =~ /perl/)
        {
           print "$ARGV: $. :$_\n";
           $pm->finish;
           goto END;
        }               
     }    
    END:
    printf ".... $dir done at %d secs!\n", time() - $start;
    $pm->finish;
}

print " all done\n";
exit;

##########################################################
sub get_sub_dirs {
 my $dir = shift;
 opendir my $dh, $dir or die "Error: $!";
 
 my @dirs = grep { -d $_ } readdir $dh;
 @dirs = grep !/^\.\.?$/, @dirs;
 closedir $dh;
 return @dirs;
}


[download]

I tend to agree that it probably would be stoppered-up by the capacity of the database server.   And let’s face it ... neither a 100MB text-file nor a 5GB database is, by today’s standards, that large.   Maybe you could make some read-only copies of the database at various places.   Maybe you could optimize the search process in the database in some useful way.   In general, I just think that trying to cluster this thing is going to be a lot of trouble, for doubtful benefit.

Clustering works really well when the workload is primarily CPU-bound and when there are no resource-contentions.   Here, both of these are not-the-case.

Edit:   BrowserUK’s subsequent recommendation to use temporary tables and a join-query, below, is in my view unquestionably the best approach to take in this case.   Now, nothing but the bulk move-in and the bulk move-out is “happening over the wire.”   The computer gets the essential job done in one step, and strictly within its own optimized world.

You'd almost certainly be better off parsing the names from the file and bulk-loading them into a temporary table within the DB. Then issue a single join to select the information you need into another temporary table; and finally dump that to a CSV for further processing if needed.

The database is likely to make far more effective use of the threading available to it that way, than having to serialise thousands of concurrent (effectively identical) queries from 64 different clients.

While I don't completely understand the work you're trying to do, I suggest checking out GNU/Parallel. It's usually a good fit for this type of stuff. It supports the unix pipeline philosophy quite well and lets you the programmer worry about the algorithm and keep it separate from the burden of scheduling tasks, distributing work, etc.