Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Re: Finding and sorting files in massive file directory

by pvaldes (Chaplain)
on Jan 20, 2013 at 21:04 UTC ( #1014332=note: print w/replies, xml ) Need Help??

in reply to Finding and sorting files in massive file directory

I have a directory with c. 2-3 million files (and growing) that I need to categorise and compress.

shell commands on the directory - find, ls etc - just hang!

Use wildcards in bash or glob in perl (ie: Fragment and move to several directories with mv a*.* /A-dir. Compress the files beginning for b with something like gzip b*.*)

  • Comment on Re: Finding and sorting files in massive file directory

Replies are listed 'Best First'.
Re^2: Finding and sorting files in massive file directory
by CColin (Scribe) on Jan 20, 2013 at 21:30 UTC
    Seem to recall when I tried something like this it failed due to too many arguments being passed to the command. ie. you can't run a million or so files through gzip command?
      If you put something like this in a while (there are files that start w/ a) loop, you should get around the issue of too many arguments. (Be sure to set increment=1 before the loop).
      mkdir dir_a.$increment mv `ls a* | head -n1000` dir_a.$increment/ let increment++
        The problem is you will get "too many arguments" from ls a* already.
        لսႽ ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ

      See the xargs command. It can help with breaking apart large command lines.

      find . -type f -name b\* -depth -2 | xargs $command_that_can_be_run_mu +ltiple_times

      Update: added example.


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1014332]
[choroba]: Will Glasgow have been out of the EU by that time already?
[LanX]: nope
hippo might venture to YAPC::EU.
[choroba]: YAPC::YAEC (yet another European country)
[LanX]: they are proposing an infinite transition phase now ...
[choroba]: infinite transition implies infinite money!
[LanX]: ... ehm ..."open ended" ;-)

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (7)
As of 2018-02-23 10:57 GMT
Find Nodes?
    Voting Booth?
    When it is dark outside I am happiest to see ...

    Results (301 votes). Check out past polls.