I see a few things that are not quite right:
my ( $wtr , $rdr , $err ) = "";
my $pid = open2 ( $wtr , $rdr , "ssh $node 'perl -e \"while(<>){ch
+omp;\$val=\$_;\$res=\`ls -l /mnt/\*/\$val\`;}\"'" );
This is a neat idea, but I can't get it to do anything but give me syntax errors. Trying to get all of the correct quotes through ssh, bash, and perl needs more far genius than I can summon at this time of night. I would just do a plain invocation of ssh and feed the "ls -l" command to it the same way you would type it at the command line.
By the way, I would leave the filehandles undefined until you pass them to open. Setting them to the null string appears to work in this case, but the builtin open() would have interpreted an undefined filehandle differently than one that was set to the null string. It would take the string value as a symbolic reference to the actual filehandle... even if it is the null string. If open2 worked the same way, $rdr and $wtr would both have been set to \"" afterwards, that is, they would both be references to a single filehandle. I wouldn't wish debugging that on my worst enemy.
print {$ssh_handles{$node}{in}} $val;
Actually, that should be {out}, not {in}. Similarly, the following bit needs to read from {in} and not {out}. That might be why you weren't seeing any output at all.
my $select = IO::Select->new($ssh_handles{$node}{out});
if(@ready = $select->can_read(4.0))
Oops! You've dug yourself a very deep hole here. Asynchronous I/O code is trickier to write than it looks at first glance, or second glance, or.... Although your code will work most of the time, it needs to loop until the input file (and error file if you're using open3) are closed. Otherwise, if ssh returns part of a line, you would read that, print it, and exit.
Trying to fix this timing race would be awkward enough, but there is a second point: error messages from the ls -l command are always going to come back from the {in} file and not the {err} file. This is because {err} is the ssh command's STDERR. Anything the ssh command gets back from the remote system will always be printed on ssh's STDOUT. The {err} would only contain error message printed by ssh itself, such as "connection refused".
I would just go with open2 and do a straightforward slurp of the whole {in} file. This does away with all the headaches of asynchronous IO.
Try this and see if it's a little closer to what you need:
use IPC::Open2;
use IPC::Open3;
use strict;
use IO::Select;
my %ssh_handles;
my @nodes = qw(x y z);
my $val = "/test/*/26/*testingTESTING*";
foreach my $node (@nodes) {
my ( $wtr, $rdr, $err );
my $pid = open2( $wtr, $rdr, "ssh", $node ) or die "Failed to open
+ ssh, $!";
$ssh_handles{$node}{in} = $wtr if ($wtr);
$ssh_handles{$node}{out} = $rdr if ($rdr);
$ssh_handles{$node}{err} = $err if ($err);
}
foreach my $node ( sort keys(%ssh_handles) ) {
print { $ssh_handles{$node}{out} } "ls -l /mnt/*/$val 2>&1\nexit\n
+";
close $ssh_handles{$node}{out};
my $data = join "", readline $ssh_handles{$node}{in};
print "[", $data, "]\n";
}
| 
