comment on

But I'm thinking that the first snippet of code walks through the data and gets the position, in bytes, for each starting number?

Correct. (But that "snippet" is a complete working program to create the index file. You did run that first didn't you?)

It gets the position (in bytes) of the start of the record (using tell),
and the first number on the line using the regex (m[^(\d+),]),
and then packs the two into a 12-byte binary record and writes it to the index file.

Also running the code you wrote as-is only produced "Not found" for me.

The first thing that comes to mind is that you never answered my question above about the sort order of your data files.

If they are not sort numerically, then you will need to sort the file; or the index file; before the binary search will work.

It's seems as if the indexing might not catch what I would like since it won't find anything I'm looking for.

I'll need a little more information to go on.

Could you run the following steps in your console and copy&paste the output (in <code></code> tags).

Substitute whatever names you gave to the two programs above for 1118102-indexer.pl & 1118102-indexer.pl below.

The test file need only be a few dozen lines; but the lines must start with numbers, and it must be sorted numerically. The 200 sample lines you posted above would be ideal. :

>perl -V

>1118102-indexer smallFile.txt smallFile.idx

>1118102-searcher -N=10 smallFile.txt smallfile.idx
[download]

If you post the output from all 3 commands, it might give some clues as to what is going on.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this

In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

In reply to Re^3: Using indexing for faster lookup in large file (PP < 0.0005s/record) by BrowserUk
in thread Using indexing for faster lookup in large file by anli_

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Think about Loose Coupling
	PerlMonks