comment on

First of all, thanks from the detailed answer. One can always expect that from you :-)

Now, indexing may indeed be my way to go. In fact, I may be needing to do it in C++. I also ran indexing, it took 14 seconds on a 3e6 line file (the file itself is ~60MB) using STL streams and 8 seconds using C file access (fgets(), ftell()). The memory consumption is pretty good, only 12 MB for these 3e6 files (in a vector). Accessing random lines with fseek() is, as expected, almost immediate.

This closes in on acceptable, and I'm starting to believe that it is indeed possible to keep it in memory. (I don't need to handle files > 4 GB).

Another problem that I forgot to mention is that the file is being updated "live" and I should keep up with it. I can get notifications, and probably just add new indexes (the file always grows, never shrinks).

In reply to Re^2: Displaying/buffering huge text files by spurperl
in thread Displaying/buffering huge text files by spurperl

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Your skill will accomplish what the force of many cannot
	PerlMonks