Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Testing for readdir failure

by Bob Cook (Acolyte)
on Apr 21, 2013 at 00:24 UTC ( #1029691=perlquestion: print w/ replies, xml ) Need Help??
Bob Cook has asked for the wisdom of the Perl Monks concerning the following question:

The following simple piece of code reads all of the entries in a directory:
if ( opendir(DIR,$somedir) ) { @x = readdir(DIR); closedir(DIR); } else { # Process an opendir error }
Clearly it might happen that the opendir works just fine, but the readdir fails due to, e.g., a hardware or network error. My question is: How can the program recognize when the readdir failed?

I have looked through all of the Perl docs I can find, searched in PerlMonks, and searched with Google.

This is not just a hypothetical question. I have a program that runs periodically and reads over 10,000,000 directories. Every once in a while, it fails to find all of some directory's entries that I know are there.

Comment on Testing for readdir failure
Download Code
Re: Testing for readdir failure
by LanX (Canon) on Apr 21, 2013 at 00:50 UTC
    Hmm ... I have problems to simulate such a problem, so my theoretical advice is to use a block eval to catch a potential error.

    And I wouldn't be surprised if $! also holds some informations, i.e. would be set in case of failure.

    Cheers Rolf

    ( addicted to the Perl Programming Language)

    update

    OK, I tested to read from a directory which was deleted in the meantime, with mixed results, sometimes the error was catched, but sometimes not ???

    use strict; use warnings; $|=1; my $dh; sub tst { my $flag=shift; print "\n\n-----"; print "Provoking error\n" if $flag; mkdir "/tmp/bla"; sleep 1; opendir $dh, "/tmp/bla" or die "$!"; rmdir "/tmp/bla" if $flag; sleep 1; eval { print scalar readdir($dh); warn "--- $!" if $! and $! !~ /^File exists/; }; warn "--- $@" if $@; } tst($_) for 1,1,0,0;

    update

    consider also catching warnings with $SIG{__WARN__}

      Hi,

      Please notice the IO operations are not written directly to the disk, but to various levels of cache. With respect of the amount of directories processed, the origin of the issue might not even be in PERL but in the operating system and the way the disk snapshot are refreshed. This would explain the mixed result. I experienced this kind of problems on MS system.

      K

      The best medicine against depression is a cold beer!
        Sure, but IMHO a cache should be transparent.

        Cheers Rolf

        ( addicted to the Perl Programming Language)

Re: Testing for readdir failure
by LanX (Canon) on Apr 21, 2013 at 02:07 UTC
    this should catch all kind of problems, that Perl is aware of!

    eval { # catch fatals use warnings FATAL => 'all'; # die instead of warn print scalar readdir($dh); }; warn "--- $@" if $@; # report error

    HTH!

    Cheers Rolf

    ( addicted to the Perl Programming Language)

Re: Testing for readdir failure
by Khen1950fx (Canon) on Apr 21, 2013 at 04:27 UTC
    Try IO::Dirent. It seems more robust.
    #!/usr/bin/perl -l use strict; use warnings; use IO::Dirent; my (@dirs) = '/'; foreach my $dir (@dirs) { opendir DIR, $dir; while ( my $entry = nextdirent(DIR) ) { print $entry->{'name'}; print $entry->{'type'}; print $entry->{'inode'}; } closedir DIR; }
Re: Testing for readdir failure
by eye (Chaplain) on Apr 21, 2013 at 04:37 UTC
    This seems to be a somewhat opaque area in the documentation.

    Have you tried testing the return value of closedir? If it reports a failure, that is probably indicative of a failure in the readdir call. Of course, I don't have a good way of testing this, either.

    Also, it is good practice to use a lexically scoped variable for the directory handle rather than a bare word.

Re: Testing for readdir failure
by Laurent_R (Parson) on Apr 21, 2013 at 10:21 UTC

    Maybe you could just try this:

    my @x; if ( opendir(my $DIR, $somedir) ) { @x = readdir($DIR) or die "$!"; closedir($DIR); }

    At least, it would give you a diagnostic if the readdir instruction fails.

    Also using the strict and warnings pragmas might help if you don't (and you should do it).

    A possible alternative would be to use the glob function, which is often more practical than the opendir/readdir combination. Perhaps something like this:

    my @x = glob "$somedir/*.*" if -d $somedir;

    would do the trick.

      From the documentation, it is not clear that readdir returns an error in $!.

      Another approach would be to stat the directory and verify that the link count matches the number of file names returned. That seems likely to catch an error, though not to identify the cause.

        > From the documentation, it is not clear that readdir returns an error in $!.

        Worse! in my tests it returns sometimes "File exists" if there's no error.

        see Testing for readdir failure

        Cheers Rolf

        ( addicted to the Perl Programming Language)

        Well it does seem to populate $! in at least some cases. I have an empty directory called "foo":

        $ perl -e 'opendir DIR, "foo" or die "$!"; @c=readdir DIR; print "@c\n +"; closedir DIR; @c=readdir DIR or die "$!"' . .. Bad file descriptor at -e line 1.

        I also got the following $! error in my quick tests under the debugger:

        DB<8> @c = readdir DIR or die "$!" Illegal seek at (eval 12)[/usr/lib/perl5/5.10/perl5db.pl:638] line 2.
Re: Testing for readdir failure
by Bloodnok (Vicar) on Apr 22, 2013 at 12:56 UTC
    Have you tried use autodie; ?

    Just a thought ...

    A user level that continues to overstate my experience :-))
      Many thanks to all of you for your thoughtful replies.

      First of all, after reading some of the replies, it became clear that I missed making one point: When the readdir fails to deliver the entire contents of the directory, there is NO warning, nor any other message, issued by Perl. Thus, I don't believe that trapping warnings, nor the warnings pragma, will help.

      I will try $!, and testing for the value of closedir, as well as testing for a "false" value of readdir (though that wouldn't work for readdir in a scalar context) and report back what I find.

      I downloaded IO::Dirent and looked at its C code. I am a complete novice at C, but it looks to me like it uses C's readdir, whose documentation seems to say that it also returns the same thing for both end-of-directory and an error. (There is a different function named readdir_r which can indicate an error, but that's not what IO::Dirent uses.) It occurs to me that the question I'm really trying to answer is: Exactly what does readdir do when it encounters a problem while reading the directory, and does it do anything that the program can recognize?

      The suggestion to stat the directory and verify that the link count matches the number of file names returned is a good one, but I can't use it because those two don't always match in the file system I'm concerned with (called AFS).

      Finally, as many have pointed out, testing any of this is clearly a problem.

        When the readdir fails to deliver the entire contents of the directory, there is NO warning, nor any other message, issued by Perl. Thus, I don't believe that trapping warnings, nor the warnings pragma, will help.

        It may be right, but if you don't try, you'll never know. Especially, if you don't turn on warnings with the warnings pragma, you're not gonna get those warnings. In my tests, when I tried to readdir on a close dir, I did not get any message unless I explicitly added the 'or die "$!"' clause to my readdir instruction.

        Consider this:

        $ perl -e  '@c=readdir DIR ; print "@c\n";'

        No warning, nothing. Compare to this;

        $ perl -w -e '@c=readdir DIR ; print "@c\n";' Name "main::DIR" used only once: possible typo at -e line 1. readdir() attempted on invalid dirhandle DIR at -e line 1.

        Turning on the warnings with the -w command line option gave me two warnings that might explain where my problem is. Or consider this:

        $ perl -e '@c=readdir DIR or print "$!"; print "@c\n";' Bad file descriptor
        > it became clear that I missed making one point: When the readdir fails to deliver the entire contents of the directory, there is NO warning, nor any other message, issued by Perl.

        It also became clear that you never clarified what exactly fails.

        How do you expect us to reproduce and catch unidentified errors?

        Are you even sure these are errors and not just race conditions because the content of a directory changed in the meantime?

        Cheers Rolf

        ( addicted to the Perl Programming Language)

Re: Testing for readdir failure
by Bob Cook (Acolyte) on May 25, 2013 at 00:15 UTC
    I inserted

    use warnings; no warnings 'uninitialized';

    into my code. (I included the no warnings because setting a variable to undef and then referring to that variable produces an uninitialized error, though what it clearly means is undefined, not uninitialized.)

    This week's run failed in the old way, and there were no warnings issued.

    I did NOT test the value of $! after the readdir because readdir does not set its value, as shown by this small piece of code executed under the debugger:

    DB<9> p opendir(D,'.') 1 DB<10> p $!+0 2 DB<11> $! = 147 DB<12> p $!+0 147 DB<13> @d = readdir(D) DB<14> p $!+0 147 DB<15>

      ???

      I did NOT test the value of $! after the readdir because readdir does not set its value, as shown by this small piece of code executed under the debugger:

      Here is a better one

      opendir D, q/./; closedir D; readdir D or die int $!, q/ /, $!; __END__ 9 Bad file descriptor at - line 3.

      perlvar#%!

      readdir STDOUT or die Fudge(); sub Fudge { join "\n", join(' ', int $!, $! ), join(' ', int $^E, $^E ), ( grep { $!{$_} } keys %! ), ' '; } __END__ 9 Bad file descriptor 6 The handle is invalid EVENT_SYSTEM_CAPTUREEND ERROR_INVALID_BLOCK EBADF EMR_SETWINDOWEXTEX EDGE_BUMP EMARCH_ENC_I17_IMM9D_SIZE_X at - line 1.
      You shouldn't do such tests in different lines of the debugger but in one line.

      DB<137> $!=undef; readdir $x or print 0+$!," ",$! 9 Bad file descriptor

      a) Each line is evaled in a different context!

      b) The debugger saves and restores special variables between evals, cause otherwise evaled code would change the error-state of the debugged program.

      Thanks BTW for discovering this!

      It shows fundamental differences between debugger and REPL functionality!

      UPDATE

      seems that I was wrong, the debugger can handle ERRNO in different lines, it's '$!=undef' which does the difference!

      DB<100> opendir $x,'.';closedir $x; DB<101> readdir $x DB<102> print 0+$!," ",$! 29 Illegal seek DB<103> $!=undef DB<104> readdir $x DB<105> print 0+$!," ",$! 9 Bad file descriptor

      For more details see Re: Testing for readdir failure (Perl-bug)

      Cheers Rolf

      ( addicted to the Perl Programming Language)

Re: Testing for readdir failure (Perl-bug)
by LanX (Canon) on May 25, 2013 at 02:50 UTC
    I finally figured out why my script testing on $! after provoked errors produced strange results.

    Seems like $! is only correct if it's reset to undef before the error occurs otherwise the last state is used.

    I couldn't find any documentation for this behaviour:

    These are my results:

    ------ Error: off --- Reset: off $! 0: --- Reset: on $! 0: --- Reset: off $! 17: File exists --- Reset: on $! 0: --- Reset: off $! 17: File exists --- Reset: on $! 0: ------ Error: on --- Reset: off $! 17: File exists --- Reset: on $! 9: Bad file descriptor --- Reset: off $! 9: Bad file descriptor --- Reset: on $! 9: Bad file descriptor --- Reset: off $! 9: Bad file descriptor --- Reset: on $! 9: Bad file descriptor

    use strict; use warnings; $|=1; my $dh; sub tst { my ($mk_error, $reset_errno)=@_; mkdir "/tmp/bla"; opendir $dh, "/tmp/bla" or warn "$!"; rmdir "/tmp/bla" if $mk_error; $!=undef if $reset_errno; my $a= scalar readdir($dh); print "\n\$! ",$!+0,": $!\n" ; } for my $mk_error (0,1) { print "\n\n------ Error: ", (qw(off on))[$mk_error] ; for my $reset_errno (0,1,0,1,0,1) { print "\n--- Reset: ", (qw(off on))[$reset_errno] ; tst($mk_error,$reset_errno) ; } }

    would be nice to know if versions > 5.10 show the same problems

    perl -version This is perl, v5.10.0 built for i486-linux-gnu-thread-multi

    UPDATE
    $! 17: File exists stems from former mkdir "/tmp/bla"; where the directory already existed.

    Cheers Rolf

    ( addicted to the Perl Programming Language)

      The same output in 5.16.0.
      لսႽ ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ

      perlvar says in the second paragraph about $!:

      Many system or library calls set errno if they fail, to indicate the cause of failure. They usually do not set errno to zero if they succeed. This means errno , hence $! , is meaningful only immediately after a failure:
        yes, but did you notice:

        ------ Error: on --- Reset: off $! 17: File exists

        I included all cases for completeness and to make it easier to find the bug.

        Maybe this is clearer:

        DB<105> $!=666 => "Unknown error 666" DB<106> $!=666;scalar readdir X or warn "$!\n" Unknown error 666 DB<107> $!=undef;scalar readdir X or warn "$!\n" Bad file descriptor

        As you can see $! needs to be reset in advance "to be meaningful".

        Cheers Rolf

        ( addicted to the Perl Programming Language)

      I have been following these contributions with interest, but I do not have the C expertise to understand some of it. So I wonder if you could advise me on these questions:

      You now refer to "perlbug errno test without localizing". Could you characterize how this bug shows up at the Perl (as opposed to the C) level? Is the answer just "readdir sometimes throws warnings and sometimes not", or can it be made more explicit?

      In addition to "use warnings", is the following the best I can do for discovering that a problem occurred during a readdir? If not, what would be better?

      $! = undef; @a = readdir(D); if ( defined($!) ) { # Process a readdir error }

      Finally, will this bug modify how "effective" this code is in recognizing the occurrence of an error on the readdir?

      Thanks very much for all your help.

        I can't neither help you on the C-level, Perl source code is very special with all its preprocessor macros.

        (UPDATE. striked nonsense, see reply)

        In the case of readdir the following should help

        $! = undef; @a = readdir(D) or process_errno();

        But in the case of readline it doesn't.

        The other advices I gave you - like catching $SIG{WARN} - are still valid,

        I will post the ticket-link after reporting the perlbug.

        Cheers Rolf

        ( addicted to the Perl Programming Language)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1029691]
Approved by LanX
Front-paged by LanX
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (10)
As of 2014-12-22 12:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (116 votes), past polls