Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Re: General perl question. Multiple servers.

by graff (Chancellor)
on Oct 06, 2007 at 15:04 UTC ( #643094=note: print w/replies, xml ) Need Help??

in reply to General perl question. Multiple servers.

Probably the easiest would be a cron job on one machine that takes a list of the machines that generate logs, harvests the log files from all of them into a daily directory, and then parses each log file in turn to do statistics and report details on "outliers".

In effect, the perl script does mechanically what you are now doing manually (you can be replaced by a perl script ;). Of course, that assumes that the 150 machines are all doing the same thing, and their results are all stored using the same path/ on each machine. Either that, or else the list of machines to scan includes all the details needed to find the log file for each one.

For doing the harvest, there's nothing inherently wrong about just running an appropriate command in back-ticks like my $log = `ssh $host cat /log/path/my_process.log` (this assumes you have public-key authentication in place, so the userid running this won't need to supply a password for each connection). If the overhead of launching a shell 150 times bothers you, you could do it like this:

use strict; use POSIX; my $today = strftime( "%Y%m%d", localtime ); mkdir "/path/harvest/$today"; chdir "/path/harvest/$today" or die $!; my @remote_machine_list = ... # (fill in the ...) my $shell_pid = open my $shell, "|-", "/bin/sh" or die $!; print $shell "cd /path/harvest/$today\n"; print $shell "ssh $_ cat /log/path/my_process.log > $_.log 2> harvest. +errlog ||". " echo $_ failed 2> harvest.errlog\n" for ( @remote_machine_list ); print $shell "exit\n"; waitpid $shell_pid, 0;
(updates: added check for success from chdir call, and added first print statement to chdir in the subshell.)

You might need to add stuff to that, like setting SIG_ALARM in case ssh hangs on a given host. On each iteration, if the "ssh $_ ..." works, its output is stored locally in "$_.log", and any stderr output is appended to the local file "harvest.errlog". But if the ssh fails, a line about that is appended to "harvest.errlog" as well.

But you also have a variety of CPAN modules in the Net::SSH domain that you might find preferable.

Replies are listed 'Best First'.
Re^2: General perl question. Multiple servers.
by dbmathis (Scribe) on Oct 06, 2007 at 23:18 UTC
    This may work, I do have public-key authentication in place already. Thanks for the information. I will let you know what I end up doing.

    After all this is over, all that will really have mattered is how we treated each other.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://643094]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (6)
As of 2021-01-25 16:03 GMT
Find Nodes?
    Voting Booth?