Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Is there a way to set max results on File::Find::Rule?

by leocharre (Priest)
on Dec 20, 2007 at 01:35 UTC ( #658035=perlquestion: print w/replies, xml ) Need Help??

leocharre has asked for the wisdom of the Perl Monks concerning the following question:

I'm using File::Find::Rule, I love it.
I set up an interface for users to seek files.
Thing is, I have a million files to search.
And I would like users to not enter any parameters, and just return a list of all files (results are paged).

With so many files, the File::Find::Rule object just keeps going and going and going... is there a way to set a limit to the iterations ? I can't find something in the docs.

  • Comment on Is there a way to set max results on File::Find::Rule?

Replies are listed 'Best First'.
Re: Is there a way to set max results on File::Find::Rule?
by runrig (Abbot) on Dec 20, 2007 at 02:01 UTC
    It sounds like you want something more like File::Find::Object (update: or File::Next), which has an iterator interface, so you can stop fetching files whenever you want to. I've never used this module so I can't comment on it much. The docs also mention other iterator-based modules, which I haven't used either.

    Alternatively, you could die in File::Find when you're done fetching (wrapping find() in an eval), but then you couldn't continue.

Re: Is there a way to set max results on File::Find::Rule?
by tachyon-II (Chaplain) on Dec 20, 2007 at 02:30 UTC

    A user interface to File::Find::Anything and a million files to search sounds like a bad combination. Have you considered the efficiency of only running File::Find periodically, dumping the results into a database of some description, and then running the search on that DB rather than on the filesystem directly. Even a flat file database would be vastly quicker than recursing the directory tree on a regular basis.

    Not suprisingly this is not a unique problem. In *nix land there is a program called (s)locate(1) which does exactly this task (recurse the tree, dump to DB, search the DB). File::Locate is a perl interface to it.

    Cheers

      In general, I hestiate to recommend a module that I wrote -- and I'm afraid I've been going through an "over-engineering" phase of late -- but if you're interested in File::Locate you might want to at least take a look at the documentation for File::Locate::Harder.

      In particular, if you look at the discussion of internals all the way down in the major section titled "NOTES", you'll see a summary of my discoveries in the world of "locate" variants.

      Note that if you want to use File::Locate you need to know where your system's "locate" database is stashed, and that varies from system to system and isn't always easily discoverable (it appears to be common for "locate"/"slocate" and "updatedb" to rely on compiled-in defaults that aren't documented in the man pages).

      Indeed, using *x slocate or indexing to a database is a good way to go. This is useful for a semi static or static set of data.

      Unfortunately, the files I am searching are a set of network mounts- multiple users, rapid changes.
      Maintaining a database of so many files would .. . require a machine solely devoted to reloading an slocate database- perpetually.
      This would simply add to the complexity of the network- it may not worth the negatives right now.

      At some point, the cpu expense of using real time disk 'find' methods will call for a separate machine to do as you mention.
      Very good comment, tachyon-II

Re: Is there a way to set max results on File::Find::Rule?
by aquarium (Curate) on Dec 20, 2007 at 02:57 UTC
    if there was some way of showing google desktop (web interface) to non-local machine...then you've got yourself a good/quick indexer + interface, all in one.
    the hardest line to type correctly is: stty erase ^H
Re: Is there a way to set max results on File::Find::Rule?
by petdance (Parson) on Dec 21, 2007 at 23:14 UTC
    The inability to stop is one of the reasons I wrote File-Next. It's the framework I built ack on.

    xoxo,
    Andy

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://658035]
Approved by Thilosophy
Front-paged by grinder
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (8)
As of 2021-06-21 07:46 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What does the "s" stand for in "perls"? (Whence perls)












    Results (98 votes). Check out past polls.

    Notices?