Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

fetchall_arrayref batches and forking

by mogmismo (Novice)
on May 07, 2013 at 19:19 UTC ( #1032527=perlquestion: print w/ replies, xml ) Need Help??
mogmismo has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to fork for each batch of a fetchall_arrayref():
my $indexers = Parallel::ForkManager->new(14); while (my $data = $r->fetchall_arrayref(undef, 5000)){ $indexers->start and next; .. do stuff .. $indexers->finish; } $indexers->wait_all_children;
But that just processes the chunks 14 times. If I reverse the order, I chunk and process the data again 14 times. What's the best way to combine these and actually fire off forks for batches of DBI results? Thanks all for helping a new junior monk here...

Comment on fetchall_arrayref batches and forking
Download Code
Re: fetchall_arrayref batches and forking
by NetWallah (Abbot) on May 07, 2013 at 22:58 UTC
    It looks like you want to process a large set of database chunks, in sets of 14.

    This fits the "single queue, multiple server" model, and BrowserUk has a thread-based implementation that can help. Please see the discussion in dynamic number of threads based on CPU utilization. Look for his code for his "threads::Q implementation".

                 "I'm fairly sure if they took porn off the Internet, there'd only be one website left, and it'd be called 'Bring Back the Porn!'"
            -- Dr. Cox, Scrubs

      Thanks, I'll take a look at those resource.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1032527]
Approved by davido
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (5)
As of 2014-10-25 06:45 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (142 votes), past polls