Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re^3: Do I need threads?

by TechFly (Scribe)
on Dec 22, 2011 at 17:55 UTC ( #944819=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Do I need threads?
in thread Do I need threads?

After reading the article there, I don't think that locking is going to help me in the long run. The idea is to speed things up, but it appears that this may introduce a lag while the output is being written, and the next process is waiting to write. That being said, I could be wrong. It may just cache the data being sent, and not written to the log without holding up the script generating the data. I will have to play with that later.

Here is what I did. Instead of writing the output from ssh to the log directly, I wrote it to a temp log, one for each thread using the captured server names for part of the file name. Then at the end of the script, I dump them all back into the main log. Still a lot of cleanup to do, but I think I am on the right track now. Thank you for pointing this out. I have no doubt you have saved me hours of confusion, and frustration!

Here is something that I missed earlier, the output being sent to the log in the fork is actually only a success or fail of the SSH call. I decided to just redirect STDOUT to the log so that I can capture the output of the command itself.

#!/usr/bin/perl use strict; use warnings; use Net::SSH qw(ssh); my %serverPid; my ($testkey, $testvalue); die("Could not open file list: $!") unless open(FHServerList, "<", "./servers"); die("Could not open log file: $!") unless open(FHlog, ">>", "./log"); while(<FHServerList>) { my $serverName = $_; chomp($serverName); my $pid = fork(); if($pid == '0'){ #Child process #open(FHTempLog, ">", "/tmp/$serverName.tmp"); open(STDOUT, '>', "/tmp/$serverName.tmp") or die ("Cannot crea +te temp log file for $serverName: $!"); open(STDERR, ">&STDOUT"); ssh($serverName, "for x in 1 2 3 4 5 6; do echo -n \$x; hostna +me; done"); exit(0) } else { $serverPid{$serverName} = $pid; } } while(($testkey, $testvalue) = each %serverPid) { waitpid($testvalue, 0); print ("Done with $testkey\n"); } seek(FHServerList, 0, 0); while(<FHServerList>) { chomp($_); my $tmpFileName = ("/tmp/$_" . ".tmp"); open(FHtempFile, "<", $tmpFileName) or die "Could not open tempfil +e: $!"; while(<FHtempFile>) { print FHlog $_; } close(FHtempFile); } close(FHServerList); exit()


Comment on Re^3: Do I need threads?
Download Code
Re^4: Do I need threads?
by Marshall (Prior) on Dec 23, 2011 at 23:54 UTC
    RE: time lag... Yes, acquiring a lock does introduce a very slight time lag...but this is very, very fast. The OS keeps track who who has what lock in a memory resident structure. To use this method, you should not acquire the lock until you are actually ready to write. That means to capture the output in a memory variable and then lock,write,unlock in quick succession rather than having the lock for the entire time that the sub-process is running. It is possible to sequence even very high I/O rates with this method because the time for each write is negligible.

    Having each child use its own individual file and then "cat" them together when all of them are finished is another way and is the way I'd do it if I was launching these tasks in the background via a shell script.

    It looks like you are using method 2, which is fine. Either way will work for your application.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://944819]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (6)
As of 2014-12-18 03:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (41 votes), past polls