Re^3: Do I need threads?

[TechFly]

After reading the article there, I don't think that locking is going to help me in the long run. The idea is to speed things up, but it appears that this may introduce a lag while the output is being written, and the next process is waiting to write. That being said, I could be wrong. It may just cache the data being sent, and not written to the log without holding up the script generating the data. I will have to play with that later.

Here is what I did. Instead of writing the output from ssh to the log directly, I wrote it to a temp log, one for each thread using the captured server names for part of the file name. Then at the end of the script, I dump them all back into the main log. Still a lot of cleanup to do, but I think I am on the right track now. Thank you for pointing this out. I have no doubt you have saved me hours of confusion, and frustration!

Here is something that I missed earlier, the output being sent to the log in the fork is actually only a success or fail of the SSH call. I decided to just redirect STDOUT to the log so that I can capture the output of the command itself.

#!/usr/bin/perl
use strict;
use warnings;
use Net::SSH qw(ssh);


my %serverPid;
my ($testkey, $testvalue);
die("Could not open file list: $!") unless
    open(FHServerList, "<", "./servers");
die("Could not open log file: $!") unless
    open(FHlog, ">>", "./log");
    
    
while(<FHServerList>) {
    my $serverName = $_;
    chomp($serverName);
    my $pid = fork();
    if($pid == '0'){
        #Child process
        #open(FHTempLog, ">", "/tmp/$serverName.tmp");
        open(STDOUT, '>', "/tmp/$serverName.tmp") or die ("Cannot crea
+te temp log file for $serverName: $!");
        open(STDERR, ">&STDOUT");
        ssh($serverName, "for x in 1 2 3 4 5 6; do echo -n \$x; hostna
+me; done");
        exit(0)
    }
    else {
        $serverPid{$serverName} = $pid;
    }
}

while(($testkey, $testvalue) = each %serverPid) {
    waitpid($testvalue, 0);
        print ("Done with $testkey\n");
    }

seek(FHServerList, 0, 0);
while(<FHServerList>) {
    chomp($_);
    my $tmpFileName = ("/tmp/$_" . ".tmp");
    open(FHtempFile, "<", $tmpFileName) or die "Could not open tempfil
+e: $!";
    while(<FHtempFile>) {
        print FHlog $_;
    }
    close(FHtempFile);
}

close(FHServerList);
exit()

[Marshall]

RE: time lag... Yes, acquiring a lock does introduce a very slight time lag...but this is very, very fast. The OS keeps track who who has what lock in a memory resident structure. To use this method, you should not acquire the lock until you are actually ready to write. That means to capture the output in a memory variable and then lock,write,unlock in quick succession rather than having the lock for the entire time that the sub-process is running. It is possible to sequence even very high I/O rates with this method because the time for each write is negligible.

Having each child use its own individual file and then "cat" them together when all of them are finished is another way and is the way I'd do it if I was launching these tasks in the background via a shell script.

It looks like you are using method 2, which is fine. Either way will work for your application.