Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re: Parallel::ForkManager using up Resources?

by cazz (Pilgrim)
on Mar 29, 2005 at 16:49 UTC ( #443180=note: print w/replies, xml ) Need Help??


in reply to Parallel::ForkManager eating up Resources?

Ouch. 140 children all with their own DB connections? And you create a new connection per child? Surely there is a better way.

First off, you should probably cut down the number of children. Most database performance numbers fall off pretty quickly when you start adding a number of children doing anything more than reads.

You should also look at using a pool of database handles. Why create and then close database connections over and over again if you don't have to? I sped up the performance of a data loading script by a few orders of magnitude by moving the dbh create/destroy outside of the main loop.
  • Comment on Re: Parallel::ForkManager using up Resources?

Replies are listed 'Best First'.
Re^2: Parallel::ForkManager using up Resources?
by guice (Scribe) on Mar 29, 2005 at 16:52 UTC

    It's a max of 5 children at once. I tried to use a pool, but couldn't figure out a way to make sure none of the children used a wrong handle. Care to share how you got past this one? I tried an array with $cnt++ % 5, but no way to guarentee each child will finish in order.

    Each child starts off opening a connection and then calls a close method which closes the connection. All that works and in order. The code is very verbose and I see that part just fine.

    -- philip
    We put the 'K' in kwality!

Re^2: Parallel::ForkManager using up Resources?
by guice (Scribe) on Mar 29, 2005 at 20:52 UTC

    Now I'm at a loss, how did you successfully create a pool of DBI handles with multiple children? I have a pool set, but as soon as my first child ends, DBI tries to clean itself up and kills all my other DBI handles with "end-of-file on communication channel during global destruction" error.

    My code (trimmed down):

    my %pool; my %dbhs; $MAX = 5; $pm->run_on_start(\&_child_start); $pm->run_on_finish(\&_child_finish); my @dbhs = start_dbh(); %pool = map { $_ => 1 } @dbhs; foreach $i ( @is ) { ($next) = grep ( defined($pool{$_}), @dbhs ); $pid = $pm->start($next) and next; $sys->set_dbh($dbhs{$next}); sleep(rand(10)); $pm->finish; } sub start_dbh() { %dbhs = map { $_ => DBI::Connect(); } 0 .. $MAX; return keys %dbhs; } Sys::set_dbh() { my $this = shift; $this->{'_dbh-child'} = shift; } sub _child_start() { my ($pid, $ident) = @_; delete($pool{$ident}); print "**** STARTED: child with PID $pid"; } sub _child_finish(){ my ($pid, $exit, $ident) = @_; print "**** FINISHED: child with PID $pid exit cide $exit"; $pool{$ident} = 1; }

    And as soon as the first child dies all my DBI handlers spew out: DBD::Oracle::db DESTROY failed: ORA-03113: end-of-file on communication channel (DBD Error: OCISessionEnd) during global destruction.

    -- philip
    We put the 'K' in kwality!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://443180]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (3)
As of 2022-12-06 23:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?