Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things

Re^12: Thread Design help

by perlCrazy (Monk)
on Sep 11, 2010 at 12:21 UTC ( #859741=note: print w/replies, xml ) Need Help??

in reply to Re^11: Thread Design help
in thread Thread Design help

thanks again and sorry for being not very clear in my reply.

we want to collect infromation from each server in every 30 minutes. for time being we will not consider servers which takes >=1 hour
We will have one group which will contain 30 data servers, if we process each server sequentially, might not be good idea. If we use thread then we can process parallely, execution will be faster and will be easy to manage.
Why not considring process ?
since forking many process( may be 100s) in one time and monitoring them will be complex. Also will consume lots of memory. But if you think forking is better option than threads, we can consider that option as well.

Replies are listed 'Best First'.
Re^13: Thread Design help
by Corion (Pope) on Sep 11, 2010 at 13:07 UTC

    Given your very vague specifications, I would very strongly consider using completely separate programs. Both, fork and threads will consume many resources and make debugging your program very hard.

    As you need to perform very different tasks (connect via ssh, connect via DBI, ...), putting all the code for these different tasks into one program makes little sense.

    Have one central program that starts the specific programs as separate children. Consider maybe Parallel::Jobs or simple open "$child |" to run your child processes in parallel.

    But before thinking about how to do things in parallel, I really, really urge you to first get things working in a serial fashion.

      Thanks for response.
      For ssh and DBI we will have two separate programs, and will be easy to maintain. for running in sequential, we will have problem.
      Example: there is poosility of delay in process for few servers and might create problem when report won't be available on time.
      I am thinking to use Thread::queue, will divide servers into 10-15 groups and kick off programm with GRP1..N as input parameter.
      1. collector. pl GRP1 ## GRP1 will contain 30-40 dataserver
      Please suggest if this approach is not efficient.

        You have two wars to fight:

        1. Retrieve information from services
        2. Parallelize tasks

        As I see it, you are currently neither clear of which has a higher priority, and what each task is to do. I really recommend that you separate the concerns. You can easily parallelize the task of information retrieval by starting programs in parallel. So do that instead of worrying about fork or threads.

Re^13: Thread Design help
by BrowserUk (Pope) on Sep 12, 2010 at 01:01 UTC

    Here's a very basic skeleton that will serve as a basis for you to tweak to do the job. It uses a prioritising subclass of Thread::Queue to 'schedule' the repeat jobs:

    #! perl -slw use strict; use threads; use Thread::Queue; use LWP::Simple; { package T::Q::O; use Data::Dump qw[ pp ]; require Thread::Queue; use threads; use threads::shared; our @ISA = 'Thread::Queue'; sub enqueue { local $^W; my $Q = shift; lock @$Q; for( @_ ) { push @$Q, $_; my $n = $#$Q; @{ $Q }[ $n, $n-1 ] = @{ $Q }[ $n-1, $n ], --$n while $n-1 and $Q->[ $n ] < $Q->[ $n - 1 ]; } cond_signal( @$Q ); } sub dump { my $Q = shift; lock @$Q; pp $Q; cond_signal( @$Q ); } } sub dbiFetch { my @args = @_; return getstore $args[ 0 ], 'nul'; } sub sshFetch { my @args = @_; return getstore $args[ 0 ], 'nul'; } sub tcpFetch { my @args = @_; return getstore $args[ 0 ], 'nul'; } sub worker { no strict 'refs'; my $Q = shift; while( my $work = $Q->dequeue ) { my( $time, $interval, $sub, @args ) = split $;, $work; sleep 1 while time() < $time; # printf "now: %.f time:$time int:$interval sub:$sub [@args]", +time(); my $result = $sub->( @args ); # print $result; $Q->enqueue( join $;, $time + $interval, $interval, $sub, @arg +s ); } } our $W //= 10; my $Q = new T::Q::O; my @workers = map async( \&worker, $Q ), 1 .. $W; while( <> ) { chomp; $Q->enqueue( join $;, time(), (map $_*60, 1,2,3 )[ rand 3 ], ( qw[dbiFetch sshFetch tcpFetch ] )[ rand 3 ], $_ ); } 1 while sleep 1;

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Thanks a lot for response, going to develop one basic module for one group of dataservers(30-40), which will have only DBI connection. Above code will be helpful to start. Will post my code once I done.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://859741]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (8)
As of 2018-03-18 14:30 GMT
Find Nodes?
    Voting Booth?
    When I think of a mole I think of:

    Results (230 votes). Check out past polls.