Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re^3: Synchronizing multiple processes retrieving files out of shared directory

by graff (Chancellor)
on Mar 21, 2014 at 03:07 UTC ( #1079159=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Synchronizing multiple processes retrieving files out of shared directory
in thread Synchronizing multiple processes retrieving files out of shared directory

I think if you were to create sibling or sub-directories on the same disk volume as the one where the depository directory exists, then the perl-internal rename function will still work (and still be atomic in the same way). Each process could create its own sub- or sister directory, and the same logic should apply.


Comment on Re^3: Synchronizing multiple processes retrieving files out of shared directory
Download Code
Replies are listed 'Best First'.
Re^4: Synchronizing multiple processes retrieving files out of shared directory
by BrowserUk (Pope) on Mar 21, 2014 at 03:15 UTC
    if you were to create sibling or sub-directories on the same disk volume as the one where the depository directory exists, then the perl-internal rename function will still work (and still be atomic in the same way).

    Is that true for NFS mounted volumes?


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      I suppose that would depend on what type of OS was being used to export the NFS volume. If it's a POSIX system, then it's using the same sort of OS-level system library call at the server, and given that old and new paths are on the same disk volume (we're just moving an inode out of one directory into another) it will be atomic, so the net result will be the same: no two separate processes will be able to rename the same file.

      That said, I just found a report that some (slightly) older versions of MacOSX were not actually using an atomic implementation of rename - but it seems that's been fixed.

      I'm working on something very similar, and I can tell you for a fact that NFS complicates things considerably.

      These resources were incredibly helpful:

      Essentially, the first time you ask NFS for a file (via stat or open) it will check the filesystem. For the next 3 seconds (by default) , any stat call is checked against the attribute cache instead of the actual file. This means that even though you check for existence (-e $file), NFS can and will lie to you.

      With a directory, the same thing is true - NFS has a filehandle cache. The first time you readdir on the directory, NFS caches that response for 3 seconds afterwards. So files can be deleted by other NFS clients and (again) NFS will lie to you.

      From the Linux NFS faq http://nfs.sourceforge.net/:

      After a file is deleted on the server, clients don't find out until they try to access the file with a file handle they had cached from a previous LOOKUP. Using rsync or mv to replace a file while it is in use on another client is a common scenario that results in an ESTALE error.

      The NFS Coding how-to has suggestions on how to work around the NFS cache as a programmer. It also has a C utility that you can download and compile on two different servers that share the NFS storage so you can determine which workarounds work and which don't.

      In general, open + close on the file and touching the directory *should* clear the caches on most Linux/Unix operating systems. VMS and Windows are a bit different though, so definitely utilize the resources above before they vanish into the dark recesses of the interwebs.

        Doesn't surprise me. I'v seen similar problems with the MS implementation of the CIFS/SMB protocol.

        In theory atomic rename works; and on a good day (probably a Thursday in June), when no one else is looking, it does. But there are no guarantees.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
        In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1079159]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (6)
As of 2015-08-01 23:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found
    past polls