Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??

Seeking help from Perl Efficiency Monks...

I need to recursively perform an action on all files in a large directory tree (tens of millions of files).

Basically, on each file I need to know the (relative) path of the file and do a 'stat' to get the inode number, the size, and the number of links.

Typically I would use File::Find but I was wondering how much overhead there was and if so would I be better off just using manual recursion with opendir/readdir/closedir and perhaps both avoid overhead and potential duplicate calls to stat (that might be buried in the find algorithm).

If, recursion with opendir is reasonably faster, does anybody have some streamlined code to offer so I can avoid "dumb" things that would slow down the recursion?

If Find is just as fast, are there any 'gotchas' I should avoid that would slow things down?


In reply to Fastest way to recurse through VERY LARGE directory tree by puterboy

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others pondering the Monastery: (6)
    As of 2015-12-01 08:08 GMT
    Find Nodes?
      Voting Booth?

      What would be the most significant thing to happen if a rope (or wire) tied the Earth and the Moon together?

      Results (796 votes), past polls