Re^3: Fresh eyes - linux::Inotify2 event delays

[kikuchiyo]

Just to offer a negative data point: we're doing something similar to your use case at $work, that is, we rely on inotify to postprocess call recordings that were finished by Asterisk, monitoring IN_CLOSE_WRITE events, just like you, and to my knowledge we've never seen such random delays in receiving events during the 3+ years this system has been in production. However, in our case the user side inotify implementation is in Go.

What kernel version do you use? That may be relevant.

[ghenry]

Found it. Our retry is in blocking mode and the CDR it is trying to find does not ever exist (normally there is a delay to them showing up). Once the retry fully expires, ->poll() does the next read on the file descriptors and picks up all the queued events. There are not many events, so no overruns (missed events) and linux just leaves them there for collection when ready. It wasn't my code :-)

[ghenry]

This is on a CentOS 7 NAS. The files are getting written to it over an NFS share, but watching on the NAS file system side. Kernel 3.10.0-1160.59.1.el7.x86_64. The python inotify has no issues it seems. We might switch to inotifywait like @choroba says and like another dept at $work uses. Or switch to C or Go. Very strange. Again, we get all the events and it feels like they are flushed to us. I might ditch retry and go to ->read. We're playing with the python version to pop UUIDs into a Redis FIFO/queue and move find_call_details() and email_recording() into separate program to speak to Redis.

[kikuchiyo]

NFS, ouch. In that case, all bets are off.

I don't have direct experience with this combination (NFS+inotify), I just remember reading about it that may work, may not work, or anything in between.

[ghenry]

Ah, but we're watching on the file system not on an NFS mount. Like I sat, Python version not missing nor are we missing events. Very weird.