I've been working on a cgi tool that takes a file and searches for a particular pair of strings and then optionally sorts the results, eliminating any duplicates. I thought I had everything working fine until someone tried to run an 80MB file through it, and his browser timed out before the searching was done.

I'm looking for suggestions on how best to approach this problem. It seems that this fellow isn't an isolated case and more people are going to need this tool to search files of this size, or even larger.

I know I could just read x lines at a time and display the results for just that section, but this presents a problem if the user requests that the results be sorted and that all duplicates are removed. The tool currently provides a count of the matches as well, something I could not easily provide with this method.

Any suggestions?

---

---

• Are you posting in the right place? Check out Where do I post X? to know for sure.
• Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:

  <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>

• Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
• Want more info? How to link or How to display code and escape characters are good places to start.