Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Efficient processing of large directory

by Elliott (Pilgrim)
on Oct 03, 2003 at 13:36 UTC ( #296228=note: print w/ replies, xml ) Need Help??


in reply to Efficient processing of large directory

Thank you all for your advice. My reading of your answers is that two approaches would help:

  • Reorganise into subdirectories as I originally thought
  • Use while instead of foreach
I have already converted to using while but I haven't had the chance to check for improvement yet. Can I have opinions on whether using both solutions together is worthwhile (pun not intended!)?

BTW, the file names are email addresses (opt-in list, no spam here I promise!!) with \W characters removed. I was planning to pick 2nd and 4th chars to name the subdirectories in order to avoid grubbiness. Any thoughts on that?


Comment on Re: Efficient processing of large directory
Select or Download Code
Replies are listed 'Best First'.
Re: Re: Efficient processing of large directory
by tilly (Archbishop) on Oct 05, 2003 at 04:04 UTC
    Switch to a dbm like DB_File instead of lots of small files. Particularly if you use a BTree, you will get much better organized usage of disk.

    But do keep text backups in case an upgrade breaks DB_File.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://296228]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (17)
As of 2015-07-07 21:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (93 votes), past polls