Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

OK, these are the assumptions and steps I made for my test based on my understanding of your requirements. First I started with a file containing official transcripts in French of a session of the European Parliament that I used to construct a file containing just one text line about 182,700 characters long:

$ wc file182730char.txt 1 28947 182729 file182730char.txt

From there, I built a 5 GB file this way: each time one identifier line with two integer random numbers between 0 and 28888, and one data line containing a copy of the above 187,000 character line, doing this 28,000 times to get my 5 GB file:

$ perl -e '$t = <>; for (1..28000) { $c = int rand 28888; $d = int rand 28888; print "> $c $d \n"; print $t}' file182730char.txt > file5gb.txt
This command took about 6 minutes to execute on my relatively old laptop. The resulting file is about 5.1 billion bytes:
$ time wc file5gb.txt 56000 810600000 5116810585 file5gb.txt real 7m54.609s user 4m3.436s sys 0m10.530s

As you can see, a simple word count (wc command) on the file took almost 8 minutes to run.

The structure of the big file is something like this:

> 12048 6179 reprise de la session [...] commission ne peut tout faire > 1024 7912 reprise de la session [...] commission ne peut tout faire > 3926 17512 reprise de la session [...] commission ne peut tout faire > 15268 6071

(with each data line above being in fact 182,729 character long.)

The idea now is to read this big file, get the two c and d random numbers on each of the identifier lines and print the c-th and d-th fields of the next data line into an output file. This can be done in just one pass over the file like this:

$ perl -ne 'if (/^> /) { ($c, $d) = (split)[1,2];} else { print join " ", (split)[$c,$d]; print "\n"};' file5gb.txt > foo.txt

Extracting the data from the big file took about 16 minutes, so about twice the duration of the simple wc command on the same file (which I think is quite good performance).

The resulting foo.txt file looks like this:

$ head foo.txt ceux cen la incapables les que grand la une en invitant que niveau d au ces consequences que un le

I do not know if my scenario is anywhere close to what you are trying to do, but that is more or less what I understood from your requirement, together with some assumptions on what the identifier might be used for on the data lines.

Your needs might be quite different, but I still hope this helps showing how you can do this type of thing in just one pass through the file.

In reply to Re: Reading HUGE file multiple times by Laurent_R
in thread Reading HUGE file multiple times by Anonymous Monk

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others meditating upon the Monastery: (4)
    As of 2019-10-19 14:29 GMT
    Find Nodes?
      Voting Booth?