How to make perl script work simltaneously on multiple objects

[slayedbylucifer]

Hello Monks, I need your expert suggestions. Let me explain what I am doing and then will present my questions:

I am working on a Perl project where my Perl script will monitor any new server being plugged into the datacenter. The moment it detects a new server in the datacenter, it will run a bunch of other security hardening Perl scripts in a particular sequence on those server to bring ‘em to our security standards.

The script which is doing this job is called Orchestrator.

Now, the whole flow is working fine however I am not able to process multiple servers at the same time.

e.g. If the 5 new servers are plugged in the datacenter, the script will pick one of them, run the flow and then pick the second one and run the flow…then pick third one and run the flow ….so on and so forth.

This is time consuming as each server customizations will take about 2 hrs to complete. So 5 servers will take 10 hrs and I don’t want this.

Question: How to make my orchestrator script to process multiple servers simultaneously?

I am a Linux sys admin who knows good amount of Perl and my knowledge of programming/development is extremely limited. I have googled about this and came with below 3 options:

Use threading in perl << I don’t know anything about threading and stu
+ff.
Use fork in perl script and fork a new instance of the same script whi
+ch will work independently on each individual server. 
Run multiple instances of the script so that each will process a new s
+erver simultaneously and independantly.

And I am absolutely not sure which one the best fit for my use case or is there anything else obvious which I missed out. Also have no idea how to practically incorporate any of above three options in my script.

Could you provide your suggestions on how to make my orchestrator work on multiple servers simultaneously.

Thanks.

[McA]

Hi,

I would go with the fork option. You seem to have a process running which detects new server instances. This would be your main loop (pseudocode):

my %found;
my %active;

# sighandler for sigchild
$SIG{'SIGCHLD'} = sub {
    my $pid = waitpid;
    delete $active{$pid};
};

while(running) {
    if(my $serverid = new_server_found()) {
        $found{$serverid} = {
            'name' => $servername,
            'otherinfo' => $otherinfo,
        };
        my $pid = fork;
        if(defined $pid) {
            # fork ok
            if($pid == 0) {
                 # this is in the child
                 do_all_stuff_necessary_for_hardening();
                 exit $rc;
            }
            else {
                # this is parent
                # You have the pid and you know that this
                # fork was concerning a server you found
                # probably you want to store this information
                $active{$pid} = $serverid;
            }
            
        else {
            # no fork => error handling
            die ("FATAL: Something went wrong");
        }

    }
    sleep(1);
}

BE AWARE: This is perlish pseudocode and should give only hints.

McA

[slayedbylucifer]

Thanks McA. Sorry for late reply. I will evaluate fork usage and will see how it goes. I not sure how much code change would it take.

[McA]

Hi,

if you fork a perl script which does have all to process the job it should be quite easy as you don't need to exec and you do have everything in hands. Anything you set up before fork is also accessible in the child. Be careful. Memory usage is multiplied.

There a some points you have to be aware of:

• Childs inherit open io handles. If you don't need them close them in childs.
• If you have a DBI handle the childs inherit it. The DESTROY handler of DBI is shutting down the connection when childs exit. So look at manpage of DBI concerning DBI and forking.
• Make a SIGCHLD handler for reaping of just for setting flags to reap in parent. waitpid is your friend. Sometimes it's nice to call the non-blocking version of waitpid.
• Be aware that the SIGCHLD handler interrupts a sleep(), so that this sleep may be much less than the amount of seconds specified.

McA

[Anonymous Monk]

Give this page a looking over. There are a few samples in here that should catch your eye. http://poe.perl.org/?POE_Cookbook

[perl-diddler]

Have you looked at Gnu's "parallel" program??

It can take a script and run it 'N' times in parallel for multiple inputs.

There's also variant called 'sem' which uses semaphores to control the number of available "processes", but

'ls' -1|parallel -j6 du -sh

would run 6 du jobs at a time on your input.

Of great interest if you really want to figure out how to do such things -- parallel is written in perl!

[slayedbylucifer]

This is good. I immediately find a lot of places where i can use parallel. Shame I wasn't aware of this (being a Linux sys admin.. ). thanks perl-diddler.

[slayedbylucifer]

Bookmarked. Wasn't aware of this. This site looks interesting. It might be of help not only in this use-case but in future perl assignments too. Thanks.

[slayedbylucifer]

By mistake, I replied to you in perl-diddler's response.