Testing for readdir failure

Bob Cook has asked for the wisdom of the Perl Monks concerning the following question:

The following simple piece of code reads all of the entries in a directory:

if ( opendir(DIR,$somedir) ) {
    @x = readdir(DIR);
    closedir(DIR);
}
else {
    # Process an opendir error
}
[download]

Clearly it might happen that the opendir works just fine, but the readdir fails due to, e.g., a hardware or network error. My question is: How can the program recognize when the readdir failed?

I have looked through all of the Perl docs I can find, searched in PerlMonks, and searched with Google.

This is not just a hypothetical question. I have a program that runs periodically and reads over 10,000,000 directories. Every once in a while, it fails to find all of some directory's entries that I know are there.

Comment on Testing for readdir failure Download Code

Replies are listed 'Best First'.
Re: Testing for readdir failure by LanX (Saint) on Apr 21, 2013 at 00:50 UTC
Hmm ... I have problems to simulate such a problem, so my theoretical advice is to use a block eval to catch a potential error. And I wouldn't be surprised if `$!` also holds some informations, i.e. would be set in case of failure. Cheers Rolf ( addicted to the Perl Programming Language) update OK, I tested to read from a directory which was deleted in the meantime, with mixed results, sometimes the error was catched, but sometimes not ??? `use strict; use warnings; $\|=1; my $dh; sub tst { my $flag=shift; print "\n\n-----"; print "Provoking error\n" if $flag; mkdir "/tmp/bla"; sleep 1; opendir $dh, "/tmp/bla" or die "$!"; rmdir "/tmp/bla" if $flag; sleep 1; eval { print scalar readdir($dh); warn "--- $!" if $! and $! !~ /^File exists/; }; warn "--- $@" if $@; } tst($_) for 1,1,0,0;` [download] update consider also catching warnings with `$SIG{__WARN__}`	[reply] [d/l] [select]
Re^2: Testing for readdir failure by Zzenmonk (Sexton) on Apr 24, 2013 at 08:43 UTC
Hi, Please notice the IO operations are not written directly to the disk, but to various levels of cache. With respect of the amount of directories processed, the origin of the issue might not even be in PERL but in the operating system and the way the disk snapshot are refreshed. This would explain the mixed result. I experienced this kind of problems on MS system. K The best medicine against depression is a cold beer!	[reply]
Re^3: Testing for readdir failure by LanX (Saint) on Apr 24, 2013 at 10:39 UTC
Sure, but IMHO a cache should be transparent. Cheers Rolf ( addicted to the Perl Programming Language)	[reply]
Re^4: Testing for readdir failure by Zzenmonk (Sexton) on Apr 24, 2013 at 11:35 UTC
Re^5: Testing for readdir failure by Bob Cook (Acolyte) on Apr 24, 2013 at 17:04 UTC
Some notes below your chosen depth have not been shown here
Re: Testing for readdir failure by LanX (Saint) on Apr 21, 2013 at 02:07 UTC
this should catch all kind of problems, that Perl is aware of! `eval { # catch fatals use warnings FATAL => 'all'; # die instead of warn print scalar readdir($dh); }; warn "--- $@" if $@; # report error` [download] HTH! Cheers Rolf ( addicted to the Perl Programming Language)	[reply] [d/l]
Re: Testing for readdir failure by Khen1950fx (Canon) on Apr 21, 2013 at 04:27 UTC
Try IO::Dirent. It seems more robust. `#!/usr/bin/perl -l use strict; use warnings; use IO::Dirent; my (@dirs) = '/'; foreach my $dir (@dirs) { opendir DIR, $dir; while ( my $entry = nextdirent(DIR) ) { print $entry->{'name'}; print $entry->{'type'}; print $entry->{'inode'}; } closedir DIR; }` [download]	[reply] [d/l]
Re: Testing for readdir failure by eye (Chaplain) on Apr 21, 2013 at 04:37 UTC
This seems to be a somewhat opaque area in the documentation. Have you tried testing the return value of closedir? If it reports a failure, that is probably indicative of a failure in the readdir call. Of course, I don't have a good way of testing this, either. Also, it is good practice to use a lexically scoped variable for the directory handle rather than a bare word.	[reply]
Re: Testing for readdir failure by Laurent_R (Canon) on Apr 21, 2013 at 10:21 UTC
Maybe you could just try this: `my @x; if ( opendir(my $DIR, $somedir) ) { @x = readdir($DIR) or die "$!"; closedir($DIR); }` [download] At least, it would give you a diagnostic if the readdir instruction fails. Also using the strict and warnings pragmas might help if you don't (and you should do it). A possible alternative would be to use the glob function, which is often more practical than the opendir/readdir combination. Perhaps something like this: `my @x = glob "$somedir/." if -d $somedir;` would do the trick.	[reply] [d/l] [select]
Re^2: Testing for readdir failure by eye (Chaplain) on Apr 21, 2013 at 12:14 UTC
From the documentation, it is not clear that readdir returns an error in $!. Another approach would be to stat the directory and verify that the link count matches the number of file names returned. That seems likely to catch an error, though not to identify the cause.	[reply]
Re^3: Testing for readdir failure by LanX (Saint) on Apr 21, 2013 at 12:16 UTC
> From the documentation, it is not clear that readdir returns an error in $!. Worse! in my tests it returns sometimes "File exists" if there's no error. see Testing for readdir failure Cheers Rolf ( addicted to the Perl Programming Language)	[reply]
Re^3: Testing for readdir failure by Laurent_R (Canon) on Apr 21, 2013 at 14:54 UTC
Well it does seem to populate $! in at least some cases. I have an empty directory called "foo": `$ perl -e 'opendir DIR, "foo" or die "$!"; @c=readdir DIR; print "@c\n +"; closedir DIR; @c=readdir DIR or die "$!"' . .. Bad file descriptor at -e line 1.` [download] I also got the following $! error in my quick tests under the debugger: `DB<8> @c = readdir DIR or die "$!" Illegal seek at (eval 12)[/usr/lib/perl5/5.10/perl5db.pl:638] line 2.` [download]	[reply] [d/l] [select]
Re: Testing for readdir failure by Bloodnok (Vicar) on Apr 22, 2013 at 12:56 UTC
Have you tried `use autodie;` ? Just a thought ... A user level that continues to overstate my experience :-))	[reply] [d/l]
Re^2: Testing for readdir failure by Bob Cook (Acolyte) on Apr 22, 2013 at 17:03 UTC
Many thanks to all of you for your thoughtful replies. First of all, after reading some of the replies, it became clear that I missed making one point: When the readdir fails to deliver the entire contents of the directory, there is NO warning, nor any other message, issued by Perl. Thus, I don't believe that trapping warnings, nor the warnings pragma, will help. I will try $!, and testing for the value of closedir, as well as testing for a "false" value of readdir (though that wouldn't work for readdir in a scalar context) and report back what I find. I downloaded IO::Dirent and looked at its C code. I am a complete novice at C, but it looks to me like it uses C's readdir, whose documentation seems to say that it also returns the same thing for both end-of-directory and an error. (There is a different function named readdir_r which can indicate an error, but that's not what IO::Dirent uses.) It occurs to me that the question I'm really trying to answer is: Exactly what does readdir do when it encounters a problem while reading the directory, and does it do anything that the program can recognize? The suggestion to stat the directory and verify that the link count matches the number of file names returned is a good one, but I can't use it because those two don't always match in the file system I'm concerned with (called AFS). Finally, as many have pointed out, testing any of this is clearly a problem.	[reply]
Re^3: Testing for readdir failure by Laurent_R (Canon) on Apr 22, 2013 at 17:53 UTC
When the readdir fails to deliver the entire contents of the directory, there is NO warning, nor any other message, issued by Perl. Thus, I don't believe that trapping warnings, nor the warnings pragma, will help. It may be right, but if you don't try, you'll never know. Especially, if you don't turn on warnings with the warnings pragma, you're not gonna get those warnings. In my tests, when I tried to readdir on a close dir, I did not get any message unless I explicitly added the 'or die "$!"' clause to my readdir instruction. Consider this: `$ perl -e '@c=readdir DIR ; print "@c\n";'` No warning, nothing. Compare to this; `$ perl -w -e '@c=readdir DIR ; print "@c\n";' Name "main::DIR" used only once: possible typo at -e line 1. readdir() attempted on invalid dirhandle DIR at -e line 1.` [download] Turning on the warnings with the -w command line option gave me two warnings that might explain where my problem is. Or consider this: `$ perl -e '@c=readdir DIR or print "$!"; print "@c\n";' Bad file descriptor` [download]	[reply] [d/l] [select]
Re^4: Testing for readdir failure by Bob Cook (Acolyte) on Apr 23, 2013 at 00:44 UTC
Re^5: Testing for readdir failure by Laurent_R (Canon) on Apr 23, 2013 at 06:19 UTC
Re^3: Testing for readdir failure by LanX (Saint) on Apr 23, 2013 at 00:52 UTC
> it became clear that I missed making one point: When the readdir fails to deliver the entire contents of the directory, there is NO warning, nor any other message, issued by Perl. It also became clear that you never clarified what exactly fails. How do you expect us to reproduce and catch unidentified errors? Are you even sure these are errors and not just race conditions because the content of a directory changed in the meantime? Cheers Rolf ( addicted to the Perl Programming Language)	[reply]
Re^4: Testing for readdir failure by Bob Cook (Acolyte) on Apr 23, 2013 at 02:45 UTC
Re^5: Testing for readdir failure by LanX (Saint) on Apr 23, 2013 at 15:18 UTC
Re: Testing for readdir failure (Perl-bug) by LanX (Saint) on May 25, 2013 at 02:50 UTC
I finally figured out why my script testing on $! after provoked errors produced strange results. Seems like $! is only correct if it's reset to `undef` before the error occurs otherwise the last state is used. I couldn't find any documentation for this behaviour: These are my results: `------ Error: off --- Reset: off $! 0: --- Reset: on $! 0: --- Reset: off $! 17: File exists --- Reset: on $! 0: --- Reset: off $! 17: File exists --- Reset: on $! 0: ------ Error: on --- Reset: off $! 17: File exists --- Reset: on $! 9: Bad file descriptor --- Reset: off $! 9: Bad file descriptor --- Reset: on $! 9: Bad file descriptor --- Reset: off $! 9: Bad file descriptor --- Reset: on $! 9: Bad file descriptor` [download] `use strict; use warnings; $\|=1; my $dh; sub tst { my ($mk_error, $reset_errno)=@_; mkdir "/tmp/bla"; opendir $dh, "/tmp/bla" or warn "$!"; rmdir "/tmp/bla" if $mk_error; $!=undef if $reset_errno; my $a= scalar readdir($dh); print "\n\$! ",$!+0,": $!\n" ; } for my $mk_error (0,1) { print "\n\n------ Error: ", (qw(off on))[$mk_error] ; for my $reset_errno (0,1,0,1,0,1) { print "\n--- Reset: ", (qw(off on))[$reset_errno] ; tst($mk_error,$reset_errno) ; } }` [download] would be nice to know if versions > 5.10 show the same problems `perl -version This is perl, v5.10.0 built for i486-linux-gnu-thread-multi` [download] UPDATE `$! 17: File exists` stems from former `mkdir "/tmp/bla";` where the directory already existed. Cheers Rolf ( addicted to the Perl Programming Language)	[reply] [d/l] [select]
Re^2: Testing for readdir failure (Perl-bug) by Corion (Patriarch) on May 25, 2013 at 08:07 UTC
perlvar says in the second paragraph about `$!`: Many system or library calls set errno if they fail, to indicate the cause of failure. They usually do not set errno to zero if they succeed. This means errno , hence $! , is meaningful only immediately after a failure:	[reply] [d/l]
Re^3: Testing for readdir failure (Perl-bug) by LanX (Saint) on May 25, 2013 at 10:26 UTC
yes, but did you notice: `------ Error: on --- Reset: off $! 17: File exists` [download] I included all cases for completeness and to make it easier to find the bug. Maybe this is clearer: `DB<105> $!=666 => "Unknown error 666" DB<106> $!=666;scalar readdir X or warn "$!\n" Unknown error 666 DB<107> $!=undef;scalar readdir X or warn "$!\n" Bad file descriptor` [download] As you can see $! needs to be reset in advance "to be meaningful". Cheers Rolf ( addicted to the Perl Programming Language)	[reply] [d/l] [select]
Re^4: Testing for readdir failure (perlbug errno test without localizing) by Anonymous Monk on May 26, 2013 at 00:20 UTC
Re^5: Testing for readdir failure (perlbug errno test without localizing) by LanX (Saint) on May 26, 2013 at 00:56 UTC
Re^5: Testing for readdir failure (perlbug errno test without localizing) by Bob Cook (Acolyte) on May 30, 2013 at 20:38 UTC
Re^2: Testing for readdir failure (Perl-bug) by choroba (Cardinal) on May 25, 2013 at 08:03 UTC
The same output in 5.16.0. لսႽ� ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ	[reply]
Re^2: Testing for readdir failure (Perl-bug) by Bob Cook (Acolyte) on May 27, 2013 at 21:04 UTC
I have been following these contributions with interest, but I do not have the C expertise to understand some of it. So I wonder if you could advise me on these questions: You now refer to "perlbug errno test without localizing". Could you characterize how this bug shows up at the Perl (as opposed to the C) level? Is the answer just "readdir sometimes throws warnings and sometimes not", or can it be made more explicit? In addition to "use warnings", is the following the best I can do for discovering that a problem occurred during a readdir? If not, what would be better? `$! = undef; @a = readdir(D); if ( defined($!) ) { # Process a readdir error }` [download] Finally, will this bug modify how "effective" this code is in recognizing the occurrence of an error on the readdir? Thanks very much for all your help.	[reply] [d/l]
Re^3: Testing for readdir failure (Perl-bug) by rdiez (Acolyte) on Jun 29, 2020 at 14:09 UTC
We may soon get a proper fix after all: https://github.com/Perl/perl5/issues/17907	[reply]
Re^3: Testing for readdir failure (Perl-bug) by LanX (Saint) on May 27, 2013 at 21:50 UTC
I can't neither help you on the C-level, Perl source code is very special with all its preprocessor macros. (UPDATE. striked nonsense, see reply) ~~In the case of `readdir` the following should help~~ ~~`$! = undef; @a = readdir(D) or process_errno();`~~ ~~[download]~~ But in the case of `readline` it doesn't. The other advices I gave you - like catching `$SIG{WARN}` - are still valid, I will post the ticket-link after reporting the perlbug. Cheers Rolf ( addicted to the Perl Programming Language)	[reply] [d/l] [select]
Re^4: Testing for readdir failure (Perl-bug) by Bob Cook (Acolyte) on May 27, 2013 at 22:20 UTC
Re^5: Testing for readdir failure (Perl-bug) by LanX (Saint) on May 27, 2013 at 23:58 UTC
Re: Testing for readdir failure by Bob Cook (Acolyte) on May 25, 2013 at 00:15 UTC
I inserted `use warnings; no warnings 'uninitialized';` [download] into my code. (I included the no warnings because setting a variable to undef and then referring to that variable produces an uninitialized error, though what it clearly means is undefined, not uninitialized.) This week's run failed in the old way, and there were no warnings issued. I did NOT test the value of $! after the readdir because readdir does not set its value, as shown by this small piece of code executed under the debugger: `DB<9> p opendir(D,'.') 1 DB<10> p $!+0 2 DB<11> $! = 147 DB<12> p $!+0 147 DB<13> @d = readdir(D) DB<14> p $!+0 147 DB<15>` [download]	[reply] [d/l] [select]
Re^2: Testing for readdir failure by Anonymous Monk on May 25, 2013 at 00:42 UTC
??? I did NOT test the value of $! after the readdir because readdir does not set its value, as shown by this small piece of code executed under the debugger: Here is a better one `opendir D, q/./; closedir D; readdir D or die int $!, q/ /, $!; __END__ 9 Bad file descriptor at - line 3.` [download] perlvar#%! `readdir STDOUT or die Fudge(); sub Fudge { join "\n", join(' ', int $!, $! ), join(' ', int $^E, $^E ), ( grep { $!{$_} } keys %! ), ' '; } __END__ 9 Bad file descriptor 6 The handle is invalid EVENT_SYSTEM_CAPTUREEND ERROR_INVALID_BLOCK EBADF EMR_SETWINDOWEXTEX EDGE_BUMP EMARCH_ENC_I17_IMM9D_SIZE_X at - line 1.` [download]	[reply] [d/l] [select]
Re^2: Testing for readdir failure (debugger) by LanX (Saint) on May 25, 2013 at 01:25 UTC
You shouldn't do such tests in different lines of the debugger but in one line. `DB<137> $!=undef; readdir $x or print 0+$!," ",$! 9 Bad file descriptor` [download] a) Each line is evaled in a different context! ~~b) The debugger saves and restores special variables between evals, cause otherwise evaled code would change the error-state of the debugged program.~~ ~~Thanks BTW for discovering this!~~ ~~It shows fundamental differences between debugger and REPL functionality!~~ UPDATE seems that I was wrong, the debugger can handle ERRNO in different lines, it's '$!=undef' which does the difference! `DB<100> opendir $x,'.';closedir $x; DB<101> readdir $x DB<102> print 0+$!," ",$! 29 Illegal seek DB<103> $!=undef DB<104> readdir $x DB<105> print 0+$!," ",$! 9 Bad file descriptor` [download] For more details see Re: Testing for readdir failure (Perl-bug) Cheers Rolf ( addicted to the Perl Programming Language)	[reply] [d/l] [select]

Back to Seekers of Perl Wisdom

update

update

UPDATE

UPDATE