Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Advice: Async Options for fire-and-forget subroutine

by mwb613 (Sexton)
on Oct 20, 2017 at 18:51 UTC ( #1201771=perlquestion: print w/replies, xml ) Need Help??
mwb613 has asked for the wisdom of the Perl Monks concerning the following question:

Thanks in advance for looking.

I took a look back through earlier questions and saw some helpful results but some of them were older and some didn't quite match my use case. I am writing a Perl module for a piece of software called OpenSIPs (a SIP server), the module itself does some message de-construction and re-construction that's not easy to do in the OpenSIPs config file itself. I have also been extended it to logging certain messages and statistics and right now my solution for that is Redis PUB/SUB. This is working fine but it is adding some latency that is not really necessary considering it is a non-essential function of the SIP transaction.

What I would like to do is, rather than wait for the Redis transaction to complete (sometimes in the range of 20-30 ms depending on where the Redis server is), just run the Redis PUB/SUB command asynchronously and not worry about the reply or even wait to get the acknowledgment in my main script. I've seen a few modules mentioned in earlier conversations here: threads, AnyEvent, Async, etc and also some non modular solutions like backticks, etc. What I'm looking for is the simplest, most lightweight way to call a Perl subroutine (passing it some info -- a JSON encoded string) and not worry about the response or whether it is successful. I'd appreciate any advice anyone is willing to give.

Thanks!

  • Comment on Advice: Async Options for fire-and-forget subroutine

Replies are listed 'Best First'.
Re: Advice: Async Options for fire-and-forget subroutine
by holli (Monsignor) on Oct 20, 2017 at 20:43 UTC
    What version of Perl are you running? There is an interesting new module called Future / Future::AsyncAwait. It requires Perl 5.24 atm though.

    With it, what you want is as easy as:
    async sub fire-and-forget { do-stuff(); } fire-and-forget(); #returns immedeatly, runs in parallel

    The author gave a talk about this at this years Perl Conference in Amsterdam1.

    1This is where Vince Vega had the time of his life.


    holli

    You can lead your users to water, but alas, you cannot drown them.

      Since I am stuck with 5.16 I tried another module, Async. It seems to work ok but there's one bit of behavior that I didn't expect (though I probably should have): if the parent script finishes, the subroutine fired by Async finishes as well (I was testing some writes after a sleep command). I guess this makes sense since the process is dying and thus everything on it's stack gets popped but I didn't comprehend this when I first started thinking about the problem.

      I haven't yet tested it within the OpenSIPs module and don't know enough about how it works to wager a guess as to whether it keeps a persistent Perl env going while it operates (I know it spawns individual worker processes so each of them might have a unique env as well). Is it an unreasonable expectation to keep looking for a module that might be able to "fire-and-forget" a subroutine that can keep running even if the parent process dies?

        In most cases, a parent process (or thread) should wait for its children to finish. In the case of processes, this (usually) cleans up zombies. In the case of thread, you have to wait until the threads are finished (or some reasonable time out) before ending the main, as this will also end the process containing all the threads.

        It is possible for child processes to "detach" from their parent. I'm pretty sure that most of the fork manager type modules in CPAN will have an option for this.

        Paraphrasing a fictitious soldier, "Fire-and-forgot is fine as long as you never forget."

      Thanks!

      Unfortunately, I am stuck on 5.16 on RHEL

Re: Advice: Async Options for fire-and-forget subroutine
by Your Mother (Chancellor) on Oct 21, 2017 at 16:15 UTC

    If you don't even care about return values, why not just use fork. You might need to be careful not to accidentally forkbomb yourself. Look at something like Parallel::ForkManager to throttle the forking if you want more control or your jobs might come in overwhelming numbers.

      Thanks Mom!

      If I were a dumb person who'd never run any sort of forking before, should I just use fork? I also see forks, perlfork, many modules including the one you listed in your post. It sounds like, based on some of the things I'm reading, that my use case -- firing and forgetting -- will create zombies which I do not want considering it will be part of a long running process.

      Also, just to clarify, the pid returned by fork() is the system PID (I'm running RHEL) or a Perl specific PID? If the former I could potentially stash the PIDs and kill them from a whole other process, no?

        Untested simplistic pseudocode that ignores a potential error point (0 == fork() is for the child side of the fork and it's not tested here). Forked processes are regular processes on *nix, just children of the original program.

        $SIG{CHLD} = "IGNORE"; while ( program_should_be_running() ) { # Might want a sleep or a usleep (from Time::HiRes) here. if ( my $pid = fork ) { # Do nothing...? This is the "permanent" parent process. # $pid is the child $$ } else { run_your_sub(); # Processes could stack up fast. } }
Re: Advice: Async Options for fire-and-forget subroutine
by stevieb (Abbot) on Oct 20, 2017 at 21:15 UTC

    My Async::Event::Interval may be of use here. It's exceptionally rudimentary in what it does.

    This software is designed to run your async event at specific intervals (every number of seconds). Here is a very brief example:

    use warnings; use strict; use Async::Event::Interval; my $event = Async::Event::Interval->new( 2, # number of seconds between execs \&callback, # code reference, or anonymous sub 'https://google.ca' # parameters to the callback ); $event->start; sleep 3; # your app does other stuff here sub callback { my ($url) = @_; print "$url\n"; }

    If you do only want it to run a single time, kill the proc after it does its work:

    sub callback { my ($url) = @_; print "$url\n"; kill 9, $$; }

    Then, if you want to spin it up manually later, just call $event->start again. If you need more arguments, append them in list form to the end of the new() call.

Re: Advice: Async Options for fire-and-forget subroutine
by salva (Abbot) on Oct 22, 2017 at 06:55 UTC
    AnyEvent and similar event-based programing modules would not solve your problem unless you rewrite your full application on top of them.

    Using a thread for handling the Redis communication is probably the more efficient approach. In your case I would try with some module implementing a queue like Thread::Queue.

    Another alternative would be to fork a slave process to work as a local buffer for the communication with Redis.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1201771]
Approved by salva
Front-paged by Corion
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (8)
As of 2017-12-18 18:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What programming language do you hate the most?




















    Results (495 votes). Check out past polls.

    Notices?