I really apologize, i dont think I'm being completely clear.

Hm. But, queues DO preserve order. That's kind of their raison d'être. (And they also 'close up the gaps' automatically!)

Generally speaking, yes they do. However in our case I really think there would be potential for things to be executed out of order using a standard queue, but not because of any fault in the queueing mechanism. Because using a typical queue, would mean we query the database before processing each node to see if resources are available-- this would immediately free up any resources and allow jobs to execute even if there were preceeding nodes that were refused and requeued. Do you follow?

If instead, (in your scenario above), you queued two items for each of your 3 users;

Again I apologize, they are being treated as individual items. So take the example again.

1. User1 issues 2 commands for machine M1
2. User2 issues 2 commands for M1, and M1 can only have 3 commands running
3. User3 issues 2 commands for M1

Using an actual queue the process would be the following

1. Query DB, see 3 resources available, Execute user1's 1st command, decrement sessions in db
2. Query DB, see 2 resources available, Execute user1's 2nd command, decrement sessions in db
3. Query DB, see 1 resources available, Execute user2's 1st command, decrement sessions in db
4. Query DB, see 0 resources available, requeue user2's 2nd command
5. Now lets say the 1st command finishes, and updates the database, incrementing its session counter
6. Query DB, see 1 resources available, Execute user3's 1st command

Now obviously, there would be ways to avoid that issue, if for instance the jobs were not incrementing their session counter off in other threads somewhere, but this would just require a rather larger to change to existing framework.

That said; I'm still not clear on why you need the shared %nodes hash, when the results from the jobs are returned to the main thread via the $Qresults?

In the actual program results are not returned to the main thread. The main thread just listens for client connection and passes the file descriptor to client threads. Results are sent back to the user, or discarded if the job was executed in the background. The %nodes hash was just to pass information back to the jobnode in the originating thread. My understanding was that when you enqueue something into the Thread::Queue, you get shared_clone of that object and thus cannot make direct modifications to the object. The hash is simply the means for returning information to the jobNode in the originating thread. Hence the

foreach my $key(keys(%$results)){
    $node->{$key} = $results->{$key};
}
[download]

The process is:

1. Main thread accepts connections, passes them to client thread
2. Client thread accepts command, creates a job node, enqueues it into the job queue (in another thread), waits until the job queue sets a job number for the node..and then some more stuff

If there is another way to get the job number from the job queue I am all ears. I thought the shared hash was a good implementation for what i needed but yeah if theres a better way to do it, happy to listen and try it out.

Thanks again for your responses!