Re^2: Speeding up data lookups

[suaveant]

The way the data is currently stored there is only ever one key... the security identifier. mmap would probably make sense, though that is only C, yes? Or is there a way to do it in perl....

If only C, then I would probably leave it as an option to look at if all Perl fails.

[sgifford]

As graff says, using mmap from Perl is straightforward using the Mmap module, which will map the file to a Perl string. You do have to be careful how you access the data; some operations will cause Perl to make a copy of the string, which is a bit of a problem with a 1GB string. substr is pretty safe, and probably most other things as long as you're careful not to write anything to the string.

I wrote up a sample grep implementation using mmap here: Re: anyone have some good mmap examples?

[graff]

MMap

You'll probably find a few nodes at PM that discuss this module.