Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

fetchall_arrayref batches and forking

by mogmismo (Novice)
on May 07, 2013 at 19:19 UTC ( #1032527=perlquestion: print w/replies, xml ) Need Help??
mogmismo has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to fork for each batch of a fetchall_arrayref():
my $indexers = Parallel::ForkManager->new(14); while (my $data = $r->fetchall_arrayref(undef, 5000)){ $indexers->start and next; .. do stuff .. $indexers->finish; } $indexers->wait_all_children;
But that just processes the chunks 14 times. If I reverse the order, I chunk and process the data again 14 times. What's the best way to combine these and actually fire off forks for batches of DBI results? Thanks all for helping a new junior monk here...

Replies are listed 'Best First'.
Re: fetchall_arrayref batches and forking
by NetWallah (Canon) on May 07, 2013 at 22:58 UTC
    It looks like you want to process a large set of database chunks, in sets of 14.

    This fits the "single queue, multiple server" model, and BrowserUk has a thread-based implementation that can help. Please see the discussion in dynamic number of threads based on CPU utilization. Look for his code for his "threads::Q implementation".

                 "I'm fairly sure if they took porn off the Internet, there'd only be one website left, and it'd be called 'Bring Back the Porn!'"
            -- Dr. Cox, Scrubs

      Thanks, I'll take a look at those resource.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1032527]
Approved by davido
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (3)
As of 2018-06-25 03:08 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (126 votes). Check out past polls.