Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Re^5: Thread terminating abnormally COND_SIGNAL(6)

by BrowserUk (Pope)
on Jul 17, 2013 at 02:27 UTC ( #1044697=note: print w/ replies, xml ) Need Help??


in reply to Re^4: Thread terminating abnormally COND_SIGNAL(6)
in thread Thread terminating abnormally COND_SIGNAL(6)

I suspect this has something to do with your simplifications, but for the life of me I cannot see what the purpose of the JobQueue is?

  1. Jobs/Nodes get queued onto it by the first set of threads.
  2. Another thread monitors it, pulls them off, sets a random value into them and then sets them into a shared hash in the main thread;
  3. But then you do nothing with that?

You seem to be getting the "results" in the main thread via the results queue; so what is the jobs queue/nodes hash for?


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.


Comment on Re^5: Thread terminating abnormally COND_SIGNAL(6)
Re^6: Thread terminating abnormally COND_SIGNAL(6)
by rmahin (Beadle) on Jul 17, 2013 at 19:14 UTC

    Ok I'll try to explain a bit further, and i apologize for the confusion, the random variable was just to test having multiple values being put into the "return value" hash. In this demo, it does indeed do absolutely nothing.

    In my main program, clients connected, and threadPool1 handles their responses. They issue commands, and can either specify those commands to be run in their same thread so they can see the output in the interactive shell, or run in the background (and get pushed to threadPool2).

    So the jobQueue. When users issue commands, their command can not always be run right away depending on their criteria and what machine in our farm they want to run it on. The queue maintains the order commands were executed in by using the JobNodes which contain information from the command they entered. The jobQueue is not REALLY acting as a typical queue, but more as a utility to block the thread that issued the command until it is it's turn (using subroutines like enqueueJob() that block until a value is set). The first case it blocks is setting the job number. After the job queue returns the job number as shown in my demo, the client thread will continue. The jobQueue will pause the command again to tell it actually what resource it should be using.

    The purpose of the JobQueue is we want to preserve the order the commands are issued in regardless of if the database is updated before the queue gets back to a job that was deferred. For example

    1. User1 issues 2 commands for machine M1
    2. User2 issues 2 commands for M1, and M1 can only have 3 commands running
    3. User3 issues 2 commands for M1

    Our queue now has (user2, user3, user3)

    The current approach, using the Thread::Queue as an array, allows us to issue one DB query at the beginning of each iteration for the state of all our resources, so we can reference that instead of querying the database for every job in the queue every time we check it. So our process is:

    1. Query database for current state and build hash
    2. check nodes
    3. repeat steps above starting at beginning of array

    This allows us to preserve the order because: if we dequeue/enqueue user2's, and one of user1's finishes and updates our database, then when we dequeue user3's command, it will see that M1 has an available resource, and run that job. That is what we want to avoid. Using it as a proper queue would not preserve the order in that fringe case without some more tinkering.

      The purpose of the JobQueue is we want to preserve the order the commands are issued in ... This allows us to preserve the order because: if we dequeue/enqueue user2's, and one of user1's finishes and updates our database, then when we dequeue user3's command, it will see that M1 has an available resource, and run that job. That is what we want to avoid. Using it as a proper queue would not preserve the order in that fringe case without some more tinkering.

      Hm. But, queues DO preserve order. That's kind of their raison d'Ítre. (And they also 'close up the gaps' automatically!)

      The problem -- I would suggest -- is that you are storing multiple requests as single items.

      If instead, (in your scenario above), you queued two items for each of your 3 users; you can now process that queue, queue-wise; re-queuing anything that isn't yet ready and discarding (moving elsewhere) anything that is complete and the queue will tale care of keeping things in their right order and ensuring that the 'wholes' get closed up, all without you having to mess around with indices trying to remember what's been removed and what not.

      Just a thought.


      That said; I'm still not clear on why you need the shared %nodes hash, when the results from the jobs are returned to the main thread via the $Qresults?


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

        I really apologize, i dont think I'm being completely clear.

        Hm. But, queues DO preserve order. That's kind of their raison d'Ítre. (And they also 'close up the gaps' automatically!)

        Generally speaking, yes they do. However in our case I really think there would be potential for things to be executed out of order using a standard queue, but not because of any fault in the queueing mechanism. Because using a typical queue, would mean we query the database before processing each node to see if resources are available-- this would immediately free up any resources and allow jobs to execute even if there were preceeding nodes that were refused and requeued. Do you follow?

        If instead, (in your scenario above), you queued two items for each of your 3 users;

        Again I apologize, they are being treated as individual items. So take the example again.

        1. User1 issues 2 commands for machine M1
        2. User2 issues 2 commands for M1, and M1 can only have 3 commands running
        3. User3 issues 2 commands for M1

        Using an actual queue the process would be the following

        1. Query DB, see 3 resources available, Execute user1's 1st command, decrement sessions in db
        2. Query DB, see 2 resources available, Execute user1's 2nd command, decrement sessions in db
        3. Query DB, see 1 resources available, Execute user2's 1st command, decrement sessions in db
        4. Query DB, see 0 resources available, requeue user2's 2nd command
        5. Now lets say the 1st command finishes, and updates the database, incrementing its session counter
        6. Query DB, see 1 resources available, Execute user3's 1st command

        Now obviously, there would be ways to avoid that issue, if for instance the jobs were not incrementing their session counter off in other threads somewhere, but this would just require a rather larger to change to existing framework.


        That said; I'm still not clear on why you need the shared %nodes hash, when the results from the jobs are returned to the main thread via the $Qresults?

        In the actual program results are not returned to the main thread. The main thread just listens for client connection and passes the file descriptor to client threads. Results are sent back to the user, or discarded if the job was executed in the background. The %nodes hash was just to pass information back to the jobnode in the originating thread. My understanding was that when you enqueue something into the Thread::Queue, you get shared_clone of that object and thus cannot make direct modifications to the object. The hash is simply the means for returning information to the jobNode in the originating thread. Hence the

        foreach my $key(keys(%$results)){ $node->{$key} = $results->{$key}; }

        The process is:

        1. Main thread accepts connections, passes them to client thread
        2. Client thread accepts command, creates a job node, enqueues it into the job queue (in another thread), waits until the job queue sets a job number for the node..and then some more stuff

        If there is another way to get the job number from the job queue I am all ears. I thought the shared hash was a good implementation for what i needed but yeah if theres a better way to do it, happy to listen and try it out.

        Thanks again for your responses!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1044697]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (9)
As of 2014-10-21 07:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (98 votes), past polls