Beefy Boxes and Bandwidth Generously Provided by pair Networks vroom
more useful options
 
PerlMonks  

How do I queue perl subroutines to a thread queue instead of data?

by static0verdrive (Initiate)
on Apr 25, 2013 at 13:37 UTC ( #1030672=perlquestion: print w/ replies, xml ) Need Help??
static0verdrive has asked for the wisdom of the Perl Monks concerning the following question:

Background: In reading how to multithread my perl script, I read (from http://perldoc.perl.org/threads.html#BUGS-AND-LIMITATIONS)

On most systems, frequent and continual creation and destruction of threads can lead to ever-increasing growth in the memory footprint of the Perl interpreter. While it is simple to just launch threads and then ->join() or ->detach() them, for long-lived applications, it is better to maintain a pool of threads, and to reuse them for the work needed, using queues to notify threads of pending work.
My script will be long-lived; it's an PKI LDAP directory monitoring daemon that will always be running. The enterprise monitoring solution will generate an alarm if it stops running for any reason. My script will check that I can reach another PKI LDAP directory, as well as validate revocation lists on both.

Problem: Everything I can find on google shows passing variables (e.g. scalars) to the thread queue rather than the subroutine itself... I think I'm just not understanding how to implement a thread queue properly compared to how you implement a thread (without queues).

Question 1: How can I "maintain a pool of threads" to avoid the perl interpreter from slowly eating up more and more memory?
Question 2: (Unrelated but while I have this code posted) Is there a safe amount of sleep (or a better method than sleep) at the end of the main program so that I don't start a thread more than once in a minute? 60 seems obvious but could that ever cause it to run more than once if the loop is fast, or perhaps miss a minute because of processing time or something?

Thanks in advance!



#!/usr/bin/perl use feature ":5.10"; use warnings; use strict; use threads; use Proc::Daemon; ### Global Variables use constant false => 0; use constant true => 1; my $app = $0; my $continue = true; $SIG{TERM} = sub { $continue = false }; # Directory Server Agent (DSA) info my @ListOfDSAs = ( { name => "Myself (inbound)", host => "ldap.myco.ca", base => "ou=mydir,o=myco,c=ca", }, { name => "Company 2", host => "ldap.comp2.ca", base => "ou=their-dir,o=comp2,c=ca", } ); ### Subroutines sub checkConnections { # runs every 5 minutes my (@DSAs, $logfile) = @_; # Code to ldapsearch threads->detach(); } sub validateRevocationLists { # runs every hour on minue xx:55 my (@DSAs, $logfile) = @_; # Code to validate CRLs haven't expired, etc threads->detach(); } ### Main program Proc::Daemon::Init; while ($continue) { my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localti +me(time); # Question 1: Queues?? if ($min % 5 == 0 || $min == 0) { threads->create(&checkConnections, @ListOfDSAs, "/var/connec +t.log"); } if ($min % 55 == 0) { threads->create(&validateRevocationLists, @ListOfDSAs, "/var +/RLs.log"); } sleep 60; # Question 2: Safer/better way to prevent multiple threa +ds being started for same check in one matching minute? } # TERM RECEIVED exit 0; __END__

Comment on How do I queue perl subroutines to a thread queue instead of data?
Download Code
Re: How do I queue perl subroutines to a thread queue instead of data?
by Random_Walk (Parson) on Apr 25, 2013 at 14:03 UTC

    If I just had two threads to signal I would create a queue for each one and just enqueue the messages when I felt the need. the thread can then block on a queue read and do nothing until it gets a message

    Updated to mention threads->create

    ... use Thread::Queue; my $conn_q = thread::queue->new() my $rev_q = thread::queue->new() sub connection_check { while (my $msg = $conn_q->dequeue) { # will block 'til it gets sign +aled # Do something } } ... # in main $conn_thread = threads->create('connection_check'); $conn_q->enqueue("Get on with it");

    $min % 55 will be zero at 55 and at 0 minutes. Perhaps two timers would do your trick

    my %timer = ( check_conn => 0, check_rev => 0, }; my %interval = ( check_conn => 5, # check conn every 5 min check_rev => 55, # and revocation at 55min }; while (1) { my $now = time; if ($now + interval{check_conn} * 60 > $timmer{check_conn} ) { #pa +ssed mark $timer{check_conn} = $now + interval{check_conn} * 60; # signal conn thread to do its work $conn_q->enqueue("Get on with it"); } # similar for check rev list # sleep 10; # or 60, or 300... }

    Cheers,
    R.

    Pereant, qui ante nos nostra dixerunt!
      Thanks for your reply! OK I think I'm starting to get it... are you saying that I never thread->create but instead only $conn_q->enqueue in the #main?

      In other words, my $conc_q = thread::queue->new() would exist at the top and no thread->create would exist anywhere because the enqueue replaces it?

        No Sorry! you still need to thread->create! Just make a thread that listens in the queue with the while (my $msg = $q->dequeue) { ... and it will wait until signalled. You can just use this to hold it up or you can check the contents of $msg and behave accordingly


        Cheers,
        R.

        Pereant, qui ante nos nostra dixerunt!
Re: How do I queue perl subroutines to a thread queue instead of data?
by sundialsvc4 (Monsignor) on Apr 26, 2013 at 01:46 UTC

    The key design-thought to keep in mind ... in any language, in any situation ... is that “a thread or process is a hamburger cook, not a single hamburger.”   Each thread goes through the same life-cycle as does any cook in a burger-joint:   wait for an order, cook it, deliver it, and then wait for another one.   And, at the end of the shift, he comes out alive.   Maybe he cooked two hamburgers that day ... maybe he cooked a hundred.

    Often, threads are specialists ... the “burger-meister,” the “fry guy” and so on.   They sit at one station, and listen to one queue.   Or they can be generalists, all grabbing from one queue and doing whatever comes.   Perl provides plenty of reliable “thread-safe queues,” and even complete workload-management systems like POE.   Your choice.

    But, whatever it is, it ought to make sense if mapped to a real-world situation like a fast food restaurant.   Don’t ask the operating system’s thread/process dispatcher to dispatch your application’s work flow, because it can’t.   The so-called “flaming arrow approach” does not work in practice.   A pool of specialized or generalized worker-bees carry out the work, and live to tell the tale.

    It is certainly a viable approach to structure the system so that Perl Objects are placed on the queue and the only thing that the worker-bee actually does is to call some execute() method implemented (in the abstract ...) by the base class.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1030672]
Approved by marto
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (5)
As of 2014-04-21 06:37 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (492 votes), past polls