Splitting one large timeout into few smaller ones..

[Eyck]

Here's the situation - I'm using Event, and am using it's timeout feature (Event::Stats). However, the procedure that's beeing watched contains variable number of steps, logically this goes like this

foreach $task (@tasks) {
 $task->do_one_task()
};

Taks that could hang include 'do_one_task' and 'foreach'. Each step should finish in under 20 seconds, but because there are up to 100 of steps involved, I need to set unreasonably large timeout value, like 2000 seconds for the whole subroutine.

This is bad because every single step can go into infinite loop, I should be able to detect this in under 30 seconds, and because I'm using aggegate values I won't know about the job hanging until 2000 seconds later.

I would like to split one big timeout value for this whole code block, into few smaller ones, essentially this means that I would like to convert this code into something like this:

foreach $task (@tasks) {
       $task->do_one_task();
        pong(); # Report back to Event::Stats that I'm
        # still alive and that it should reset it's timer
};

Alternatively I would need to use something like coroutines, but I think this would be way beyond my skill, ie:

 call_with_timeout(mysubroutine,$low_timeout);

where mysubroutine would recall the place where it last finished, ie

  $self->MagicallyRecallState(); # and jump to the correct
          #codeblock
 foreach $sth (@sth) {
    $self->SaveState() && return;
   $sth->method();
};

Which approach is the best (and I'm not sure if this coroutine-like solution is at all possible with perl)?

update

  for (0..int(rand(1000))) {
    sleep(int(rand(10)));# this takes at most 10s  
 };

[Roy Johnson]

Could you fork each task and wait for it, setting a timeout in the wait?

[Eyck]

Tasks cannot be reordered. Lets look at something like this
1. Start processing/new transaction
2. Get message from spool(requires contacting remote spool)
3. verify signature (requires contacting remote keyservers)
4. uncompress/convert/split
5. Send required files(remote...)
6. Request additional parts(remote..)
7. End transaction

Or, better yet, think of pop3 client,

1. Authenticate
2. Get list of new mails
3. Get mail body (repeat) ...

Imagine what happens when you implement it like this:

 auth();
 @uidls=getuidls();
 foreach $uidl (@uidls) {
    fork || getmail($uidl);
 };

getting back to my original problem, you can easily put timeout protection inside getuidl and getmail, but what you created is either instant DOS against your provider, and yourself (try that code with 1000s of mails to download..).

Not to mention that now you have to devise a clever way for generating for example unique filename for every mail you're downloading and few similar headaches.

Of course if you're trying to write reasonable code, and use just one connection for downloading those mails, you need to devise a clever scheme of locking and passing socket fd around to your children etc...

[Roy Johnson]

I didn't suggest you reorder tasks. You fork and wait. If the wait times out, you kill the child and error out. You only have one spawned sub-process at a time.

Update: you'd have to rig up the timeout yourself, depending on what your system supports. You want to poll and sleep until the process completes, or until you've waited long enough for it. Or you could set an alarm, do a wait, and if the wait returns, unset the alarm.

[tall_man]

You might get better results with POE. There is an example in the POE cookbook of managing many slow, forked processes.

[Eyck]

Do you think there's some simple way of translating this:

 Event->idle(min=>$mint,max=>$maxt,
 cb=> [$sth,"method"],
 max_cb_tm=>$timeout,);

into POE? Or would I have to scrap all the code I have written so far and start from the beggining?

Also, the problem is not exactly with processes being slow, but with multiplying timeouts. Look:

sub vicous {
#....
  for (0..int(rand(1000))) {
    sleep(int(rand(10)));# this takes at most 10s  
 };
#...
}

All this code is wrapped in single timeout, ie

 alarm(10000);
 vicious();
 alarm(0);

Above written as example, not actual code, because as I said, I'm using Event and not the actual alarms, but, what I'm trying to do is:

sub vicious {
#...
  for (0..int(rand(1000))) {
     sleep(int(rand(10)));
#....
     alarm(0);#reset alarm, we're OK
     alarm(10);#reset alarm..
  }; 

#...
alarm(10);
vicious();
alarm(0);

This theoretically would result in every 'task' operation being kept under 10s.

[tall_man]

One thing you should know about POE is that it uses cooperative multitasking. Your example with "sleep" would not work as you wanted under POE without some changes. As it says here:

As noted above, a POE application is a single-threaded process that pretends to perform asynchronous actions through a technique called cooperative multitasking. At any given time, only one subroutine inside a POE application is executing. If that one subroutine has a sleep 60; inside of it, the entire application will sleep for 60 seconds. No alarms will be triggered; no actions will occur. Smaller blocks of code mean that POE gets a chance to do other actions like cleaning up sessions that are shutting down or executing another event.

That's why I suggested POE could manage your tasks if each was in a child process. It may be more trouble to adapt to POE than it's worth to you in this particular case, but if you were going to adopt Roy Johnson's forking idea, it would simplify the process management and communication.

[Roy Johnson]

How about the alarm solution, without the forking?

$SIG{ALRM} = sub {
    die "Timed out"; # Or whatever
}
foreach $task (@tasks) {
    alarm 30;
    $task->do_one_task();
};
alarm 0;
$SIG{ALRM} = 'DEFAULT';

[Eyck]

Yeah, that would be exactly what I was asking about, I just hoped for something less dramatical (without die), but that's good enough, I can always wrap all that code in eval and just catch this 'die'.

Thanks.

Interestingly this works well with Event, I was affraid it wouldn't (Event uses timeout handling written in C, but it seems like it's alarm based). Here's the final code:

use Event;
use Event::Stats;
Event::Stats::enforce_max_callback_time(1);

sub callback {

 for (1..int(rand(999))) {
 alarm(10);
 sleep 2;#relatively long running tast
 };
};

$Event::DIED = sub {
        Event::verbose_exception_handler(@_);
        Event::unloop_all();
}; # just die

Event->idle(min=>1,max=>2,cb=> \&callback,max_cb_tm=>10,);
Event::loop();

[BrowserUk]

You say that the tasks must be run in order, but also, if any given task takes too long, you can abort it it and move onto the next task.

Could you start the next task once you reach it's time limit without waiting for the previous task to be aborted?

[Eyck]

There are multiple jobs, they are supposed to run periodically (or in response to some events), every jobs consists of multiple tasks, so, when I abort the tasks, job gets cancelled, thus I can move on to another job (or even re-start the current one).

I don't know how I could move to another job without aborting the current one, that would require something coroutinish Coro.

[BrowserUk]

Coro, fork, threads. Basically, any form of multitasking that allows your main process/thread (scheduler?) to initiate several concurrent jobs.