perlquestion
gizmo_mathboy
<p>I'm try to speed up some data collection on a bunch of Windows pc's.</p>
<p>Basically I need to parse some info out of the the Event Log. I then munge some data from a bunch of files and then pass data back in an array of arrays.</p>
<p>My current programs takes about a minute per pc to run and I am testing about about 70pc's at the moment and could potentially need to run this on 1,000's.</p>
<p>Here's a stripped down version of the code.</p>
<code>
my ($file_listing_pcs) = @ARGV;
#slurp in file
my $data_ref;
while (<$file>) {
my $pc = $_;
if ( ping_it($pc) ) {
($data_ref) = process_pc($pc,$data_ref);
}
else {
next;
}
}
# post process data
exit(0);
sub process_pc {
my ($pc, $data_ref) = @_;
my ($event_log_data) = get_event_log_data($pc);
my $dir_path = qq(//$pc/c\$/temp);
($data_ref) = process_dir($event_log_data,$dir_path,$data_ref)
return($data_ref);
}
sub get_event_log_data {
my ($pc) = @_;
# play with Win32::EventLog
return($event_log_data);
}
sub process_dir {
my ($event_log_data,$dir_path,$data_ref) = @_;
# read directory
# get list of files to read
# read files
# plunder data
# munge it about
return($data_ref);
}
</code>
<p>After spending time googling, reading the docs and PerlMonks I just need to use Coro (I don't think I need AnyEvent but I'm not sure).</p>
<p>The biggest time suck is dealing with the Event Log.</p>
<p>My first naive attempt to use Coro is to just change the while loop that processes the list of pc's to be:</p>
<code>
my @pids;
while (<$file>) {
my $pc = $_;
if ( ping_it($pc) ) {
push @pids, async{ ($data_ref) = process_pc($pc,$data_ref) };
cede;
}
else {
next;
}
}
$_->join for @pids;
#data profit?
</code>
<p>This didn't change the time it took to process my test horder at all.</p>
<p>I can't recall the last time I event pondering doing anything event related let alone thread related.</p>
<p>Any and all guidance much appreciated.</p>
<p>gizmo</p>