Message Queue library

[pwagyi]

Greeting Monks! I would like to seek advice from senior monks on this matter. I am developing automatic data re-processing system. Basically it is pretty similar to TCP data re-transmission.

A process will generate a bunch of data di (d1,d2,d3...dn), and it must be transferred to receiver process. Sender sends di+1 if receiver reply di (with OK) otherwise, Sender will re-send di again. Basically Receiver should receive data in order (d1,d2,..dn) and sender should at the end received replies with (OK) in order (d1,d2,..dn) Note: Sender and receiver are different processes (potentially across network) and there are time gaps between data generation (d1, 1/2 hour ,d2 , 1 hr, d3 ..)

My plan currently is as follow: There will be persitence queue with some sort of locking mechanism. I have found some modules (http://search.cpan.org/~revmischa/Data-Queue-Persistent-0.13/lib/Data/Queue/Persistent.pm) but it does not seem to provide locking the queue.

OnDataGenerationEvent(Data d) - 
    queue.add(d, {state => PENDING_TRANSFER} )
    process(queue)

OnDataReceivedEvent(Data reply) - 
   lock(queue) {
      d = dequeue()
      if reply.status = OK then
          # no Op
      else # status is NAK
          # add back to front of queue
          queue.add_front(d,{state => PENDING_TRANSFER})
      endif
          
   }
   process(queue)

sub process(queue)
   lock(queue ){
       d = dequeue(queue)
       if d.state = PENDING_TRANSFER then 
            transfer(d)
            
            queue.add(d, {state => PENDING_REPLY } )
       endif
   }

[haukex]

I think that a decent message queue should guarantee "transmission" of messages - under what conditions would you expect to lose messages? Maybe you could expand on your explanation, because I also don't understand what's wrong with just using a TCP connection?

[Anonymous Monk]

Why does a queue need a lock? Isn't the entire idea that it's a flexible buffer where readers wait for something to read and writers maybe wait for space in which to write?