Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Finding and sorting files in massive file directory

by pvaldes (Chaplain)
on Jan 20, 2013 at 21:04 UTC ( #1014332=note: print w/ replies, xml ) Need Help??


in reply to Finding and sorting files in massive file directory

I have a directory with c. 2-3 million files (and growing) that I need to categorise and compress.

shell commands on the directory - find, ls etc - just hang!

Use wildcards in bash or glob in perl (ie: Fragment and move to several directories with mv a*.* /A-dir. Compress the files beginning for b with something like gzip b*.*)


Comment on Re: Finding and sorting files in massive file directory
Re^2: Finding and sorting files in massive file directory
by CColin (Scribe) on Jan 20, 2013 at 21:30 UTC
    Seem to recall when I tried something like this it failed due to too many arguments being passed to the command. ie. you can't run a million or so files through gzip command?
      If you put something like this in a while (there are files that start w/ a) loop, you should get around the issue of too many arguments. (Be sure to set increment=1 before the loop).
      mkdir dir_a.$increment mv `ls a* | head -n1000` dir_a.$increment/ let increment++
        The problem is you will get "too many arguments" from ls a* already.
        لսႽ ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ

      See the xargs command. It can help with breaking apart large command lines.

      find . -type f -name b\* -depth -2 | xargs $command_that_can_be_run_mu +ltiple_times

      Update: added example.

      --MidLifeXis

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1014332]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (9)
As of 2014-12-18 05:36 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (42 votes), past polls