Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling

Re: Receiving data asynchronously in a long-run process

by Joost (Canon)
on Sep 21, 2009 at 18:15 UTC ( #796588=note: print w/replies, xml ) Need Help??

in reply to Receiving data asynchronously in a long-run process

The worker should enqueue the job for processing, then continue processing its current job.

Unless your worker is actually capable of doing more than one job at a time (by, for instance, forking off other processes to do that) there is no reason to fetch jobs ASAP. In the simpler case where jobs are processed sequentially you generally only want to get a new job once the old one it finished. But that will possibly block the issuer if the pipe fills up.

Also, unless you're talking about really simple systems, you generally want to have multiple workers (and possibly multiple issuers) running at the same time, possibly at different machines. There are generic systems for doing the message queueing in that situation, they're called message queues.

The message queue will recieve jobs (messages) and set up routing and queues for multiple receivers as you need. That simplifies the workers to a more or less pure "fetch, process, repeat" loop. Take a look at activemq or rabbitmq for example. (Note: activemq takes some work to configure, but supports STOMP as the protocol which is really straightforward. To understand rabbitmq you need to understand the AMQP specs and use POE - and neither are completely trivial**)

I can only think of one other options to handle this IPC problem; make the worker multithreaded with one pipe management thread and one work thread. This seems like a pretty heavyweight solution to a pretty simple problem.
POE works well for these situations, probably better than threads, though I wouldn't describe it as lightweight.

Also, unless you're comfortable with dropping jobs when crashes occur and running everything on a single machine or thread, your problem isn't simple. :)

edit: **) for now, you may also want my forks of the AMQP modules, here and here (you need both).

  • Comment on Re: Receiving data asynchronously in a long-run process

Replies are listed 'Best First'.
Re^2: Receiving data asynchronously in a long-run process
by Annirak (Novice) on Sep 21, 2009 at 19:54 UTC

    Wow, joost, that's a lot of really great information. My intent was originally to write a library which would work much like Thread::Pool::Simple, so you're absolutely right; I was planning to use multiple workers. I had intended this to be a local-machine-only library, but the possibility of handling off-machine jobs for particularly granular tasks has a lot to recommend it too.

    For a local-machine library, I suspect that activemq is probably heavier than necessary. For a grid computing library, it's probably a great solution. I don't think I need quite that much power, but since I do want to release this library, perhaps the flexibilty is worthwhile.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://796588]
[1nickt]: This is not to suggest that Karl should remove his slogan. Just thoughts of mine... Trust me I believe I am very much more radical than most, and if I had a slogan for every gross injustice in my sig, there would be no more bytes!
Discipulus redskins .. well native american..
[1nickt]: So if I was to offer any editorial advice it would be to (1) make it endure time passing (2) make it positive (3) make it about you and your beliefs
[1nickt]: (Disicpulus The new best choice is maybe "First People"
[karlgoethebier]: Discipulus: This is called "Volksverhetzung" in German which means
[karlgoethebier]: incitement of the people and it is prohibited by law
[1nickt]: The best sign I have sign in a protest march was carried by a kid of about 10 years "If you build a wall my generation will tear it down."
[1nickt]: ... sign I have seen ...
[1nickt]: Now that one made me hopeful! First time not angry in a long time when I saw that kid.
[Discipulus]: i propend for removing: why? because we are so few that we must find i minimal common divisor, this is certainly Perl not our (anyway private) thougths. And i say this still wondering because i love a lot freedom of expression. And i say this not for roho

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (9)
As of 2017-06-22 12:10 GMT
Find Nodes?
    Voting Booth?
    How many monitors do you use while coding?

    Results (519 votes). Check out past polls.