Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Limitations to chmod and performance

by stefan k (Curate)
on Nov 12, 2001 at 16:46 UTC ( #124815=perlquestion: print w/ replies, xml ) Need Help??
stefan k has asked for the wisdom of the Perl Monks concerning the following question:

Fellow Monks,
my SysAd is in the process of writing a permission-fix-a-scripttm. and he is going to do that in perl (thereby replacing the old and ugly shell script). Since there is a lot of files to check (several GB) this should be as rock-solid as possible and as fast as possible.

He uses File::Find, of course :-).

One question that arose is whether it would be better to create a giant list of files and pass them to the chmod command and rely on it's implementation or to call chmod for every single file. (Probably the middle-solution of creating groups of a certain size himself is stoopid). This is the question of performance.

The second question is whether chmod has a limit. Is there a maximum number of files/dirs that you can pass to chmod?

Well, and since this seems to be a quite common problem I may add to his questions whether this not already exists somewhere (permission fixing system, including exception lists, group settings and more). A quick search in the Monastery yielded no perfect results for me.

Regards... Stefan
you begin bashing the string with a +42 regexp of confusion

Comment on Limitations to chmod and performance
Select or Download Code
Re (tilly) 1: Limitations to chmod and performance
by tilly (Archbishop) on Nov 12, 2001 at 17:24 UTC
    Working one file at a time reduces the possibility that something changes in the filesystem between when you see the file and change the permission. It also means that there is no upper limit on how many files you can handle.

    Working with a list of files and letting chmod do the loop is probably faster than looping over chmod statements. But if your list of files is very large, you could run out of memory. (Very unlikely, but there is a theoretical limit there.)

    Personally I would suggest working with whatever style fits the surrounding code the best. With File::Find that would be one file at a time, just do the chmod in the callback.

    (Actually I would be inclined to go to the shell and just do a chmod -R ..., but it sounds like you have been down that route...)

      Working with a list of files and letting chmod do the loop is probably faster than looping over chmod statements. But if your list of files is very large, you could run out of memory. (Very unlikely, but there is a theoretical limit there.)

      I'll furthermore add that, with many implementations of 'chmod' the number of args is limited,
      and you'll probably hit a 'arg list too long' error way before you exhaust your memory...
      (For example on my linux box I'm limited to around 12900 files as arguments...)

      To my mind you'd definitly better use the one-at-a-time strategy...

      Note : This problem also occurs with a lot of others file related tools : rm, ls...

      "Only Bad Coders Code Badly In Perl" (OBC2BIP)
Re: Limitations to chmod and performance
by c-era (Curate) on Nov 12, 2001 at 17:57 UTC
    We just replaced a script that fixed permisions (basicaly chmod -R) because it was too slow. I found it was faster to read the permisions on every file, and then only do a chmod to the files that needed it. I should note, our system is running on average 85% disk utilization (and yes it is raided). Just try a couple of different scripts, and benchmark them to see what works best for your system.
Re: Limitations to chmod and performance
by Fletch (Chancellor) on Nov 12, 2001 at 19:51 UTC

    Technically, the underlying chmod(2) system call only takes two arguments (the mode and the path). So no matter how many filenames you pass to perl's chmod it's going to devolve into multiple system calls somewhere underneath anywhere.

      On a linux box, this is easily illustrated by the following experiment (better carried on the /tmp directory):

       strace -e trace=file perl -e 'chmod 0777, "a".."z"' 2>&1 | tail -26

      One will see 26 chmod system calls.

      -- stefp

Re: Limitations to chmod and performance
by Zaxo (Archbishop) on Nov 12, 2001 at 20:57 UTC

    You might consider system 'find'.

    $ find /the/dir -name \*.dat -not -perm 644 -exec chmod 644 \{\} \;
    find2perl can translate such a command to pure perl, but it doesn't understand +signed notation for the -perm any-bit argument (version dated 1998/04/07, easy to fix that).

    Whatever solution you use, you can prune statting all regular files in a dir whose mtime is prior to the last run. chmod acts on the inode, not the file, so the directory times are what counts.

    After Compline,
    Zaxo

      Actually such a script is replaced. And I wouldn't know how to elegantly and efficiently have a list with exceptions put in that code. But thanks anyway (the hint with mtime and last run is a good one (just like the prior postings contained good hints - thank you, Monks! :)

      Regards... Stefan
      you begin bashing the string with a +42 regexp of confusion

      Using the command find instead of File::Find is not a bad thing per se. But as you do it, you will fork a chmod command per matching file. Since you are concerned by performance issue, let us say it may be better to optimize by bundling as much files as possible in an exec command by using xargs:

      find /the/dir -name \*.dat -not -perm 644 | xargs chmod 644

      To be on the safe side and to deal correctly with weird file names (like windoze filenames with blanks), it is even better to use the null character as a file separator:

      find /the/dir -name \*.dat -not -perm 644 -print0 | xargs -0 chmod 644

      -- stefp

Re: Limitations to chmod and performance
by {NULE} (Hermit) on Nov 12, 2001 at 23:09 UTC
    Hi,

    We had a slew of directories that were created with the wrong permissions on a few of our servers recently. In an effort to make our admin's life a little easier I whipped up this bulk permissions changing script.

    It doesn't do exactly what you want, but it might give you a head start. It recurses through the directory trees offering to change any to mode 755 that are set something else. A nice feature is the verbose logging and the creation of a bash friendly undo script when it's done.

    Parts of this code were from when I was a younger, less-wise monk so I don't use Getopts or File::Find - but this isn't terribly complicated so it doesn't hurt anything. If you are going to start your own from scratch you'd be wise to check those out. It also doesn't bother with checking the owner since root happened to own all files in this particular case.

    Good luck! Click on the READMORE tag for the code.

    {NULE}
    --
    http://www.nule.org

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://124815]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (7)
As of 2014-10-23 05:18 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (124 votes), past polls