Beefy Boxes and Bandwidth Generously Provided by pair Networks
Come for the quick hacks, stay for the epiphanies.
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??

I'm trying to build a worker process that accepts and processes jobs. I'm getting hung up on how to accept the jobs: I'd like the worker to be able to accept jobs without having to check for them periodically. The way I have implemented it currently is this:

  1. The job issuer spawns a worker, and connects to it via a pipe. my $pid=open($pipe,"| $worker");
  2. The job issuer freezes a job object (or hash or whatever)
  3. The job issuer calculates the length of the frozen object, then sends print $pipe "$len\n$job"
  4. The worker picks this job up by: $len=<STDIN>;chomp $len; read(STDIN,$data,$len);$ref=thaw($data);
  5. The worker should enqueue the job for processing, then continue processing its current job.
  6. goto 2.

Here's the catch: For the second or later iteration, the worker may be in the middle of a long run process, so there's no way to know how long the worker might take to service STDIN. While I could just let data sit in the pipe, I'm not sure how reliable that is. More to the point, it makes it harder (impossible?) for the worker to collect information about pending jobs.

So I got the bright idea of signaling the worker when data has been written to its input pipe. Here's the worker:

#!/usr/bin/perl use strict; use Storable qw(freeze thaw); BEGIN{ $SIG{USR1}=\&drdy; } my @datastore=(); sub drdy { print "Signal!\n"; my $d=<STDIN>; print $d."\n"; push @datastore,$d; $SIG{USR1}=\&drdy; } while (1) { sleep 1; print shift @datastore if (scalar @datastore); }

This worker gets stuck in &drdy after the first SIGUSR1. This leads me to believe that I can't read from STDIN in a signal handler. The job issuer sets {my $fh = select $pipe;$|=0;select $fh} so the problem is not that the communication is buffered. The entire transaction should be in the worker's input pipe before the job issuer signals the worker.

I can only think of one other options to handle this IPC problem; make the worker multithreaded with one pipe management thread and one work thread. This seems like a pretty heavyweight solution to a pretty simple problem.

Is multithreading the way to do this?


In reply to Receiving data asynchronously in a long-run process by Annirak

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others rifling through the Monastery: (5)
As of 2024-05-21 00:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found