There is a good reaason why search engines don't allow full regex searches--they are simply too slow. You'd be better only allowing keyword searches.
But, if you decide to try the big file route, a couple of things that will help are:
- When concatenating your files, remove all the newlines so that each file becomes a single line prefixed by the path information.
This makes searching for phrases that span lines much simpler and much, much faster.
If you need to obtain position information, re-search the actual files once you've located them from the master file.
- If you have multiple cores available, split the master into as many parts as you have cores and search them in parallel.
If you can distribute these parts across different spindles, you'll gain the maximum advantage of the parallelism. Otherwise, you may not see much benefit due to head thrash.
None the less, by making each file a single long line in the master, the ratio of cpu to IO will be greatly improved.
- If your OS/filesystem allows you to override the default IO buffer size, increasing it to encompass the average (or maximum) line length may yield benefits.
- Make your master file(s) contiguous if your filesystem allows that.
If you can limit a first pass to keywords only, then building an inverted index would be the fastest way of trimming the dataset. You could then allow a full regex search to be run on the subset of documents.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
|