Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Re: Finding and sorting files in massive file directory

by pvaldes (Chaplain)
on Jan 20, 2013 at 21:04 UTC ( #1014332=note: print w/ replies, xml ) Need Help??

in reply to Finding and sorting files in massive file directory

I have a directory with c. 2-3 million files (and growing) that I need to categorise and compress.

shell commands on the directory - find, ls etc - just hang!

Use wildcards in bash or glob in perl (ie: Fragment and move to several directories with mv a*.* /A-dir. Compress the files beginning for b with something like gzip b*.*)

Comment on Re: Finding and sorting files in massive file directory
Replies are listed 'Best First'.
Re^2: Finding and sorting files in massive file directory
by CColin (Scribe) on Jan 20, 2013 at 21:30 UTC
    Seem to recall when I tried something like this it failed due to too many arguments being passed to the command. ie. you can't run a million or so files through gzip command?
      If you put something like this in a while (there are files that start w/ a) loop, you should get around the issue of too many arguments. (Be sure to set increment=1 before the loop).
      mkdir dir_a.$increment mv `ls a* | head -n1000` dir_a.$increment/ let increment++
        The problem is you will get "too many arguments" from ls a* already.
        لսႽ ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ

      See the xargs command. It can help with breaking apart large command lines.

      find . -type f -name b\* -depth -2 | xargs $command_that_can_be_run_mu +ltiple_times

      Update: added example.


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1014332]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (4)
As of 2015-11-30 02:36 GMT
Find Nodes?
    Voting Booth?

    What would be the most significant thing to happen if a rope (or wire) tied the Earth and the Moon together?

    Results (756 votes), past polls