Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re: Queuing in multithread context

by Anonymous Monk
on Jan 20, 2015 at 10:58 UTC ( #1113855=note: print w/replies, xml ) Need Help??


in reply to Queuing in multithread context

I try to implement the Thread::Queue module, but the provided example and the other resources I have been reading only consider a defined amount of queued tasks (when this may vary in my case).

What does that mean?

Its unclear what your diagram looks like, but I think Thread::Queue is workable like this

#!/usr/bin/perl -- use strict; use warnings; use threads stack_size => 4096; use Thread::Queue; my %serverQ; my $resultQ = Thread::Queue->new; my @workers = map { my $workQ = Thread::Queue->new; threads->create( \&Worker, $workQ, $resultQ ); } 1..10; ... while( @targets ){ ... if( my $q = $serverQ{ $host } ){ $q->enqueue( [ ## 'Foo::bar::deployRoutine', $onearg, $twoarg, $threearg , 'deployRoutine' => $targetServer, $deploySource, $deployDest, "true", ] ); ... FreeFreeWorkers( \%serverQ ); ## check empty Qs, unassociate from +a host ... $_->join for threads->list; exit( 0 ); sub Worker { my( $workQ, $resultQ ) = @_; while( defined( my $jobDesc = $workQ->dequeue ) ) { my( $package, $function, @args ) = @$jobDesc; require Module::Load; Module::Load::load( $package ); $resultQ->enqueue( [ $function->( @args ) ] ); } return; }

Replies are listed 'Best First'.
Re^2: Queuing in multithread context
by Anonymous Monk on Jan 20, 2015 at 11:02 UTC
    Bah, typo, idea is
    ### workQs , workerQs my @workers = map { my $workQ = Thread::Queue->new; threads->create( \&Worker, $workQ, $resultQ ); $workQ; } 1..10;
Re^2: Queuing in multithread context
by Hardin (Novice) on Jan 20, 2015 at 11:28 UTC

    Hi!

    Thanks for taking time to answer, sorry if I didn't make myself clear, my aim is to achieve queueing of my sub "deployRoutine" so that each copy task is executed after another and not sent alltogether.

    I have two main loops, the first loop through "targets" (ie: server where I need to copy files/directories) and the second embbeded is to loop through "deployments" which are several separated copy tasks.

    Lets say I have 3 servers : srv1, srv2 & srv3, I need to copy on each of those some remote content to local contents: copy1, copy2 & copy3 with

    • copy1: source=\\some\remote\content1 dest=C:/some/local/content1
    • copy2: soure=\\some\different\remoteconte dest=C:/another/local/destination
    • copy3: ... (you get the idea)

    I want the copy tasks (I said 3 but that could be 4, 5 or more, this can vary) sent to each servers one after another while currently they are sent at the same time since I create the thread its directly executed to the server.

    Right now I'm trying to understand your code as it's bit unclear to me.

    Cheers :)

      my aim is to achieve queueing of my sub "deployRoutine" so that each copy task is executed after another and not sent alltogether.

      If you want to execute the subroutines sequentially; don't use threads!

      Just call the subroutine from within your loop; and they'll execute sequentially with no conflicts. Simples!


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
      In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
Re^2: Queuing in multithread context
by BrowserUk (Pope) on Jan 20, 2015 at 11:11 UTC

    Please ignore this. I had a brain fart! Please don't advocate Thread::Queue. Its broken.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
    In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

      Please don't advocate Thread::Queue. Its broken.

      Why? I learned to advocate it from your postings :)

        Sorry. I brain-farted. I've scratched my post above.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
        In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1113855]
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (5)
As of 2017-11-22 00:06 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In order to be able to say "I know Perl", you must have:













    Results (312 votes). Check out past polls.

    Notices?