Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Safely create new directory

by elTriberium (Friar)
on Jul 25, 2012 at 19:08 UTC ( [id://983719]=perlquestion: print w/replies, xml ) Need Help??

elTriberium has asked for the wisdom of the Perl Monks concerning the following question:

Say we have multiple systems that all have an nfs-mounted filesystem. All of them want to create a new directory under a common dir, e.g. /mnt/home/volume. I know that there's the O_EXCL mode to sysopen which is guaranteed to fail if a file already exists, but is there a similar way to create a directory?

I though about using this pseudocode:

my $dir = "/mnt/home/volume/"; my $i = 0; while(1) { $i++; my $success = 1; sysopen (FH, $dir . "$i", O_RDWR|O_EXCL|O_CREAT) or $sucess = 0; last if $success; } # Now it should be safe to create dir $i

This looks kind of crappy, is there a better way to do that or a module which already provides that functionality?

Thanks!

Replies are listed 'Best First'.
Re: Safely create new directory (mkdir, NFS)
by tye (Sage) on Jul 25, 2012 at 19:37 UTC

    mkdir is already atomic on local file systems. creat() with O_EXCL is not always atomic over NFS (depends on NFS version and kernel version, just when talking about Linux). I find indications in both directions as to whether mkdir() is atomic over NFS; so much so that I gave up trying to find the real answer.

    Your approach can't work because there is no atomic "delete the file 'foo' and create the directory 'foo'" operation.

    If it turns out that you have an NFS environment with atomic file creation semantics but mkdir isn't atomic (or you just can't determine whether it is or not), then you could use something similar to your approach.

    1. Increment $i
    2. Atomically create "reserve.$i" file
    3. next if that failed
    4. mkdir "$i" (which should never fail)

    Note that you can't delete the "reserve.$i" file unless you know nobody is going to try to create a new directory after that. You could go with a more complicated algorithm if you want to delete the "reserve.$i" files.

    1. Increment $i
    2. next if -d "$i" (optional but avoids some thrash)
    3. Atomically create "reserve.$i" file
    4. next if that failed
    5. if -d "$i" then unlink "reserve.$i", next
    6. mkdir "$i" (which should never fail)
    7. unlink "reserve.$i"

    - tye        

      Thanks, that helps! About your point that there's no atomic "delete the file and create the dir" operation. I understand that, but my point was to just leave the file there. Sure it's a dirty solution, that's why I was looking for better solution. Just using "mkdir" sounds good, if it's atomic then that will do the trick. I saw similar issues with O_EXCL not being guaranteed on older Linux kernels, but it was mentioned that it should be safe from 2.6.5 onwards.
Re: Safely create new directory
by BrowserUk (Patriarch) on Jul 25, 2012 at 19:52 UTC

    (WARNING: Old, fuzzy, possibly inapplicable and out-of-date memory from the distant past!)

    On HP-UX 10.something, we successfully used something like:

    my $dir = "/mnt/home/volume/"; my $i = 0; ++$i until mkdir "$dir$i"; ## $i contains the dir successfully created

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

      Thanks all!

      Together with the other information in this thread that mkdir is in fact atomic this will do.

        nformation in this thread that mkdir is in fact atomic

        I don't believe that atomicity of mkdir was ever in doubt as far as local filesystems are concerned.

        The question, I thought, was about whether it remained atomic for NFS mounted directories.

        My point in posting was -- on ancient HP/UX, where mkdir was tested and found atomic on NFS-mounted drives -- not just that that simple form of loop was sufficient to obtain a unique directory, but actually that it is far more likely to be reliable than the two-stage mechanisms that create files first, then directories.

        Tendered with the clear caveat of the ancient state of my knowledge, I fail to see that creating a file -- just another name in the directory space -- is any more likely to be atomic than creating a directory.

        But ... that is based on logic alone; not current, OS-specific knowledge.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?

Re: Safely create new directory
by moritz (Cardinal) on Jul 25, 2012 at 19:24 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://983719]
Approved by moritz
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others learning in the Monastery: (2)
As of 2024-03-19 06:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found