Re^3: Testing for readdir failure

[LanX]

Sure, but IMHO a cache should be transparent.

[Zzenmonk]

Definitely! My issue looked very similar and was the result of a bug in the "fsflush" process (name of the process in Solaris). This process flushes all buffers to the disk. It is also responsible to provide a consistent snapshot of the disk + buffers state via an API to the other components of a system.

I currently have a similar issue on a Linux box. The mixed hard disk snapshot is not always consistent. Sometimes the buffers are not properly flushed to the hard disk and I loose the solid state part of the storage. This matter has the same behavior as this case: no error at all at any level.

So provided you have a unix box try the following:

# Prior to any opendir() statement issue:
system("sync");
# This will flush all buffers to the disk and refresh the disk snapsho
+t
# Next issue your opendir statement
opendir(...);
# Start your processing

If your issue disappears it is most probably an OS problem and PERL can not really help but you have a solution.

K

[Bob Cook]

This is a very interesting suggestion. And, the program IS running on Linux machines. Do you know if there are analogous problems in the case of reads (you talked about problems with writing, which I am not doing)? And, is there any code for suppressing that condition analogous to your "sync" command for writes?

Of course, though this might illuminate my current problem, it does not solve the more general question: How does a Perl program discover that readdir could not execute correctly due to, for example, a hardware disk error?

[Zzenmonk]

Hi,

First this is just a hypothesis. I am not a low level operating system IO programmer, so I do not really know. I really had issues with read, but on a MS platform. My code simply did not see new files, despite the fact I opendir() / close DIR for any loop the code does. The origin of the problem was effectively the IO system of MS Windows Server. I also experience similar read and write problem on my Linux laptop. In both case the sync command, respectively the MS equivalent solves the issue.

The sync command impacts also the read operations. Wen you issue the opendir() command your script receives a consistent snapshot of the disk+buffer state. This snapshot must remain constant during the time the script processes the directory content or the array of the directory content would vary in size.

In this context imagine millions of opendir(), close DIR operations. The burden of the operating system in order to provide your script with consistent buffers+disk snapshots becomes quite high. So what will usually works might fail in your case. By issuing sync commands the dirty IO buffers, meaning the ones changed and not visible yet in your snapshot are written to the disk. Accordingly the read operation is simplified because less dirty buffers are available and this might solve the issue.

Why PERL does not detect the problem, I do not know. In the cases I experienced I could not find any error message at any level. I suppose the operating system does not return an error code to PERL unless maybe put in a debugging mode.

If this is the origin of the problem, there is no solutions to it and no way to avoid it. The management of dirty buffers is a problem in all system. I know this from my database experience.

K