Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

decoupling processes via queues: advice wanted

by morgon (Priest)
on Apr 19, 2009 at 15:47 UTC ( #758584=perlquestion: print w/replies, xml ) Need Help??

morgon has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

in my current project we will have several cooperating Perl-processes running on several machines.

What we are currently thinking about is not having these processes talk directly to each other but decoupling them via a queuing middle-tier in which each processes would communicate indirectly via (persistent) queues.

We are running on Linux and use Oracle for the backend.

The question is now how to implement such a queuing infrastructure.

I can currently see four ways for us to achieve this:

1) Roll your own.

2) Use Spread.

3) Use database-tables are queues.

4) Use Oracle Advanced Queuing.

Currently I am trying to evalute options 2-4 and would welcome any feedback on specific pros and cons that more experienced monks see.

Personally I quite like Spread (so far - I don't have a lot of experience with it), however I am also seeing the advantages of Oracle Advanced Queuing as it would permit us to have a process dequeue a message, do it's business-logic, update some database-state and finally enqueue new messages all in one database-transaction.

So could you please provide some enlightenment?

Many thanks

  • Comment on decoupling processes via queues: advice wanted

Replies are listed 'Best First'.
Re: decoupling processes via queues: advice wanted
by rcaputo (Chaplain) on Apr 19, 2009 at 18:33 UTC
Re: decoupling processes via queues: advice wanted
by dHarry (Abbot) on Apr 19, 2009 at 18:19 UTC

    You´re not to clear on why you want to do it. If you´re having problems with your current implementation you might want to investigate the cause. Under the assumption that decoupling indeed is a good idea:

    Sounds to me like you want a message queue. I would not try to implement functionality like this yourself. My experience is limited to MQ Series (nowadays called WebSphere MQ). If I understand correctly Oracle Advanced Queuing is a similar product. So this is one alternative to research.

    Instead of going for broke it might be wise to do a small proof of concept. Messaging is not a silver bullet. I don´t know Oracle AQ, I assume you have to spend money on it. There are several free/open source products on the market. See Message queue for somepointers. I advice to do a small PoC to see if it will work for you

      decoupling indeed is a good idea
      I believe it is a good idea.

      What we want is that processes to not depend on other processes to be running. In our scenario the processes don't know about each other, they know only about the queues they use to either receive messages from or to post messages to.

      Sounds to me like you want a message queue.
      Evidently - that's what I said in the title, or didn't I?

      My experience is limited to MQ Series
      I've used MQ and would like to use it again but we don't have it and won't get it. If it was an option I would have listed it as number 5.

      Oracle Advanced Queuing is a similar product. So this is one alternative to research.
      This is why I posted this message...I had hoped for some feedback of people who have used it.
      I don´t know Oracle AQ, I assume you have to spend money on it
      It is part of Oracle. If you pay for Oracle you get AQ for free - that's why it is an option for us.

      There are several free/open source products on the market
      For the time being I would like to restict the discussions to the above listed options...

      But many thanks anyway.

        I believe it is a good idea.

        It might be a great idea but I cannot judge that based on the information you provide. I don’t know what the Perl scripts are doing and why they communicate. I would just like to say that not all synchronous communication can be changed to asynchronous.

        In general asynchronous messaging allows for more parallelism because of the non-blocking nature but it also introduces issues like what happens if a message cannot be delivered? Or maybe the receiver needs to return some transaction id to the sender? etc.

        I have seen home grown messaging middleware developed at a time when messaging products were not available. Nowadays I would take an existing product rather than reinventing wheels. Dedicated Message Oriented Middleware normally takes care of much of the complexity.

        If Oracle AQ is available at no extra cost why not give it a try? I read up on Oracle AQ, and it is implemented in database tables (your option 3).

        Cheers,
        dHarry

Re: decoupling processes via queues: advice wanted
by andreas1234567 (Vicar) on Apr 20, 2009 at 12:14 UTC
    The Message queue (wikipedia article) lists 5-6 other open source alternatives to the ones already listed.
    --
    No matter how great and destructive your problems may seem now, remember, you've probably only seen the tip of them. [1]
Re: decoupling processes via queues: advice wanted
by mrons (Beadle) on Apr 21, 2009 at 04:04 UTC
    I've implemented a queuing infrastructure using IPC::DirQueue

    Some of the other persistent queue implementations on CPAN don't allow multiple processes accessing queues. IPC::DirQueue can work with multiple processes and even multiple hosts via NFS.

    Also the CPAN queue implementations I've looked at don't have the idea of a job on the queue being active, they only do push and shift. An active job stays on the queue until you say "I've finished with that job". Other processes don't pick up the job while active.

    Great module. It may fit your needs if you don't want to dive into oracle queues.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://758584]
Front-paged by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (5)
As of 2022-11-29 03:42 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?