limiting children

[aijin]

I'm working on a server at the moment. Currently, it receives requests from the clients and pushes them into a list. Then, it forks off a new process to process the next request popped off the list.

What I want it to do, is watch how many children are running and only start a new process if there are under x children.

This would be, from what I've read, easy as pie if I had Parallel::ForkManager available, but I don't, so it's not turing out to be that easy.

Anyone have any design advice or tips for me? I'm new at this whole server thing and having a rather rough time of it.

-aijin

[dragonchild]

In this case, I'd suggest get Parallel::ForkManager from CPAN and make it "easy as pie". :-)

------
We are the carpenters and bricklayers of the Information Age.

Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

[aijin]

Ah, well if it were up to me, I would.

-aijin

[dragonchild]

Go get it, install it in your home directory, and do a use lib '/my/directory/where/I/install/stuff';

------
We are the carpenters and bricklayers of the Information Age.

Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

[fokat]

I would do this:
• Put the code that does the fork() in a function that takes care of it. This must check the list of pending things to do, and try to fork() a new child for each item in the queue until a preset limit. Also, make sure that this function increases a global counter each time that forking is done.

• Create a handler for SIG_CHLD that upon a succesful waitpid(), reduces the global counter and attempts to reschedule by calling the function in the prior item.

• In your event loop, after pushing a thing to do in the queue, call the function in the first bullet.

This ensures that your code will be calling the scheduler whenever there is something to do and/or a child terminates.

I agree, however, that it is probably cheaper to try to get a module that already does all this correctly, as my suggestion would be reinventing the wheel.

Now somebody that knows more about the Perl's internals than me, could clarify on the risk of getting your script to dump core. (Note that I have some code like this that does a few millions operations such as those a day and never had any occurrence).

The risk would come from the fact that the scheduler function would be performing list management on the arrival of a signal, which is an asynchronous event. The signal might arrive while the scheduler is manipulating the list of tasks when called by the event loop. In this case, a reentrant call to the scheduler would attempt to modify the list with a potentially inconsistent data structure.

In practice, however, you should not worry about this :)

Good luck.

[merlyn]

Please read what I said in "No excuses about not using CGI.pm".

If you can't install a module, at least grab the source to Parallel::ForkManager, cut and paste it into your program, and vet the code. Claim it was your own, and you'll get done 20 hours earlier than you would have if you have to reinvent this already proven wheel.

I really, really, really do not understand this "module-phobia". It's OPEN SOURCE people. Use, reuse, contribute, patch. It's the way.

-- Randal L. Schwartz, Perl hacker

[jlongino]

It doesn't seem like it would be that complicated. Here is a very generalized approach. Of course you'll have to fill in the subs as required:

use strict;
while (1) {
  my $run_ct = how_many_running();
  my $wait_ct = how_many_waiting();
  foreach (1..(10-$run_ct)) {
     launch_next_waiting();
  }
  sleep 20;
}

sub how_many_running{}
sub how_many_waiting{}
sub launch_next_waiting{}

Hope this helps.

"Make everything as simple as possible, but not simpler." -- Albert Einstein

[merlyn]

For a nice one that reuses the kids, see the first and second part of a recent LM column I wrote about doing that.

-- Randal L. Schwartz, Perl hacker

[belg4mit]

Proc::Queue seems useful for this...