Is there a definitive module for efficiently searching a collection of text files?

[nysus]

I’m writing a utility to go through my personal perl lib to ensure consistency across files. For example, I will use it to check each file has a comment containing the file’s path. So the utility will be going through and repeatedly searching a lot of text files.

There will be hundreds of documents searched repeatedly for certain strings of text. Rather than reinvent the wheel, I hunted around for a module that would help ensure the search was done efficiently both in terms of time and resources used so I don’t have to burden myself with worrying about the many small details. For example, the easiest thing for me to do would be to load all files into memory and simply search, write them back out to disk, load them all back into memory, search, save, repeat. Though this is easy, it’s obviously not efficient and I was hoping to find a module that would smartly determine how to handle this problem.

But I have searched around a bit and have come up empty. Does anyone have any recommendations?

[Discipulus]

Hello nysus,

> ..efficiently ..

did you tried MCE::Grep ? See also other examples on github

L*

[sn1987a]

You may want to checkout the ack utility and the module behind it

[nysus]

Ah, this looks very promising. Thanks!

[Anonymous Monk]

Do you have any requirement that grep doesn't satisfy?

[nysus]

Two things: getting something off the shelf that just works and avoiding agonizing over the minutiae of grep commands. And taking care of details like file locking, fast searching and making multiple changes to files in memory to make things more efficient.

I've started to patch a solution together myself with Path::Iterator::Rule for selecting files of interest and then creating a custom file object that will handle in memory search and replaces and file locking. It's probably overkill for what I need but I'd rather make sure I try to do it right. I won't be able to beat grep when it comes to raw searching speed, but I'm not too concerned about that. I'm only dealing with a few hundred files at most.

[Marshall]

You said: "I'm only dealing with a few hundred files at most."

I would forget about optimizing. Do what is the easiest thing for you to implement. I think you are looking at well less than a minute, maybe even just some ten's of seconds. Also, since these are only relatively small text files (software source), the file will probably be in disk cache if you need to read it again.

[Fletch]

Not Perl but ripgrep is really fast.