Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

I've envisioned a design for a listening service at $work that I'd like to implement, but I'm not sure how to do it right. I've iterated over it in my head, but I'm not sure if my ideas are the best approaches. I'm asking for some feedback.

First let's start with what I'd like to accomplish:

  • I'd like a daemon Perl service to listen for instructions on a unix socket or sockets (concurrency is needed)
  • The instructions will come as pairs of hostnames and system commands. After a hostname/command are passed in, the client receives the ID of its new task, and disconnects.
  • The listening service will establish a remote connection to the hostname provided, and run the command.
  • The commands are anticipated to take a long time to run, producing a steady stream of output (such as `iostat -dxk 3`)
  • Line by line, the service will stuff the output of the running command into a Thread::Queue or a file (haven't decided yet; memory could be a factor)
  • Every 3 to 4 seconds the client reconnects to the listening service and supplies the ID of its task, asking for whatever output has accumulated since it last connected
  • The service sends whatever is in its queue back to the client
  • At any time the service can terminate the long-running command based one one of two criteria:
    • Nobody has checked on the status of task ID N for $timeout seconds
    • A client connects to the service and issues a kill command while providing a task ID; the service then kills that task and ends its queue

The threading comes in now: I need the service to be able to process at least 20 connections in parallel, without making clients wait for a turn to run their command. I need a supervisor thread to monitor run times for all tasks and kill them off if they haven't been checked on in $timeout seconds.

The classic supervisor-worker thread model might not work here, which is where I'm stumped. I'd have to have three types of threads, not two: 1) the supervisor, 2) the task runners, 3) the listeners. Why not forks? I don't want to use forks, because I need each listening thread to be able to know about all running tasks via threads::shared in-memory variables (I'm not going to be using a database to keep track of running tasks).

I've recently gained a new respect for threading in Perl since my success with it in the recent hackathon, and I'd like to use those lessons learned in order to achieve success in this next endeavor.

What do you think?


A mistake can be valuable or costly, depending on how faithfully you pursue correction

In reply to Help designing a threaded service by Tommy

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?

What's my password?
Create A New User
Domain Nodelet?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (7)
As of 2023-03-24 09:14 GMT
Find Nodes?
    Voting Booth?
    Which type of climate do you prefer to live in?

    Results (60 votes). Check out past polls.