Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery

Message Queue library

by pwagyi (Scribe)
on Jun 21, 2018 at 01:49 UTC ( #1217087=perlquestion: print w/replies, xml ) Need Help??

pwagyi has asked for the wisdom of the Perl Monks concerning the following question:

Greeting Monks! I would like to seek advice from senior monks on this matter. I am developing automatic data re-processing system. Basically it is pretty similar to TCP data re-transmission.

A process will generate a bunch of data di (d1,d2,d3...dn), and it must be transferred to receiver process. Sender sends di+1 if receiver reply di (with OK) otherwise, Sender will re-send di again. Basically Receiver should receive data in order (d1,d2,..dn) and sender should at the end received replies with (OK) in order (d1,d2,..dn) Note: Sender and receiver are different processes (potentially across network) and there are time gaps between data generation (d1, 1/2 hour ,d2 , 1 hr, d3 ..)

My plan currently is as follow: There will be persitence queue with some sort of locking mechanism. I have found some modules ( but it does not seem to provide locking the queue.

OnDataGenerationEvent(Data d) - queue.add(d, {state => PENDING_TRANSFER} ) process(queue) OnDataReceivedEvent(Data reply) - lock(queue) { d = dequeue() if reply.status = OK then # no Op else # status is NAK # add back to front of queue queue.add_front(d,{state => PENDING_TRANSFER}) endif } process(queue) sub process(queue) lock(queue ){ d = dequeue(queue) if d.state = PENDING_TRANSFER then transfer(d) queue.add(d, {state => PENDING_REPLY } ) endif }

Replies are listed 'Best First'.
Re: Message Queue library
by haukex (Chancellor) on Jun 21, 2018 at 07:54 UTC

    I think that a decent message queue should guarantee "transmission" of messages - under what conditions would you expect to lose messages? Maybe you could expand on your explanation, because I also don't understand what's wrong with just using a TCP connection?

Re: Message Queue library
by Anonymous Monk on Jun 21, 2018 at 16:39 UTC
    Why does a queue need a lock? Isn't the entire idea that it's a flexible buffer where readers wait for something to read and writers maybe wait for space in which to write?
    A reply falls below the community's threshold of quality. You may see it by logging in.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1217087]
Approved by marto
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (10)
As of 2019-10-15 12:29 GMT
Find Nodes?
    Voting Booth?