Re^2: Testing for readdir failure

[Bob Cook]

in reply to Re: Testing for readdir failure
in thread Testing for readdir failure

Many thanks to all of you for your thoughtful replies.

First of all, after reading some of the replies, it became clear that I missed making one point: When the readdir fails to deliver the entire contents of the directory, there is NO warning, nor any other message, issued by Perl. Thus, I don't believe that trapping warnings, nor the warnings pragma, will help.

I will try $!, and testing for the value of closedir, as well as testing for a "false" value of readdir (though that wouldn't work for readdir in a scalar context) and report back what I find.

I downloaded IO::Dirent and looked at its C code. I am a complete novice at C, but it looks to me like it uses C's readdir, whose documentation seems to say that it also returns the same thing for both end-of-directory and an error. (There is a different function named readdir_r which can indicate an error, but that's not what IO::Dirent uses.) It occurs to me that the question I'm really trying to answer is: Exactly what does readdir do when it encounters a problem while reading the directory, and does it do anything that the program can recognize?

The suggestion to stat the directory and verify that the link count matches the number of file names returned is a good one, but I can't use it because those two don't always match in the file system I'm concerned with (called AFS).

Finally, as many have pointed out, testing any of this is clearly a problem.

[Laurent_R]

When the readdir fails to deliver the entire contents of the directory, there is NO warning, nor any other message, issued by Perl. Thus, I don't believe that trapping warnings, nor the warnings pragma, will help.

It may be right, but if you don't try, you'll never know. Especially, if you don't turn on warnings with the warnings pragma, you're not gonna get those warnings. In my tests, when I tried to readdir on a close dir, I did not get any message unless I explicitly added the 'or die "$!"' clause to my readdir instruction.

Consider this:

$ perl -e  '@c=readdir DIR ; print "@c\n";'


No warning, nothing. Compare to this;

$ perl -w -e  '@c=readdir DIR ; print "@c\n";'
Name "main::DIR" used only once: possible typo at -e line 1.
readdir() attempted on invalid dirhandle DIR at -e line 1.

Turning on the warnings with the -w command line option gave me two warnings that might explain where my problem is. Or consider this:

$ perl -e  '@c=readdir DIR or print "$!"; print "@c\n";'
Bad file descriptor

[Bob Cook]

Wow, I had no idea that warnings had different "levels" and that by default, some weren't issued. I don't think I want to enable ALL warnings, but the io bunch looks appropriate.

Thanks for the suggestion.

[Laurent_R]

I think that pretty much everyone on this forum will agree that you should always "use strict;" and "use warnings;" for every single program that you write, except possibly for programs which have strictly less than two lines. This is increedibly useful to find syntax and programming errors.

[LanX]

> it became clear that I missed making one point: When the readdir fails to deliver the entire contents of the directory, there is NO warning, nor any other message, issued by Perl.

It also became clear that you never clarified what exactly fails.

How do you expect us to reproduce and catch unidentified errors?

Are you even sure these are errors and not just race conditions because the content of a directory changed in the meantime?

[Bob Cook]

> It also became clear that you never clarified what exactly fails.

The program runs weekly. Many weeks, it "fails" on a small percentage of directories. Those directories were last changed years ago. And yet, on this run, according to the results of readdir, there were some things missing that were present on previous runs, AND are present when I go to look at them after the failure. (And by the way, each week it fails on a different small subset of directories.)

> How do you expect us to reproduce and catch unidentified errors?

I don't know how to reproduce it. That's a real problem. I was looking for possible side effects from readdir that the program could check for. And some of the suggestions have been quite helpful, including yours.

> Are you even sure these are errors and not just race conditions because the content of a directory changed in the meantime?

I hope the first part of this text above answers this. Most of the directories haven't changed.

[LanX]

Thanks "missing files" is an information.

There are any possibilities, it could be your filesystem, it could be a perl bug, it could be a stupid race condition.

For instance I had very confusing results in the tests of my first reply, deleting the opened directory before reading the files produced different results in consecutive runs... maybe I was to tired to nail down the problem.

Anyway running your code now, with all messages (Fatals, Warnings and $!) catched and logged should hopefully give you a hint for suspicious signals.