And I wouldn't be surprised if $! also holds some informations, i.e. would be set in case of failure.

update

OK, I tested to read from a directory which was deleted in the meantime, with mixed results, sometimes the error was catched, but sometimes not ???

use strict;
use warnings;

$|=1;
my $dh;

sub tst {
  my $flag=shift;

  print "\n\n-----";
  print "Provoking error\n" if $flag;
  
  mkdir "/tmp/bla";
  sleep 1;
  opendir $dh, "/tmp/bla" or die "$!";

  rmdir "/tmp/bla" if $flag;
  sleep 1;


  eval {
    print scalar readdir($dh);
    warn "--- $!" if $! and $! !~ /^File exists/;
  };
  warn "--- $@" if $@; 
}

tst($_) for 1,1,0,0;
[download]

update

consider also catching warnings with $SIG{__WARN__}