comment on

If I understand the crux of the matter is: You need to look up data in a file that is too big to hold in RAM?

Perhaps you can build an index for the file that is smaller and then use that to look into the file. If performance is an issue memoizing your lookup may help. But then again premature optimization and all that...

You may also want to put it into a DB. SQLite is great way if you just want to create it then use it for lookups. Not so great if you need concurrent access, then you may need a real DB server

Cheers,
R.

Pereant, qui ante nos nostra dixerunt!

In reply to Re: Parallel::Forkmanager and large hash, running out of memory by Random_Walk
in thread Parallel::Forkmanager and large hash, running out of memory by mabossert

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


We don't bite newbies here... much
	PerlMonks