Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Re^3: Read Some lines in Tera byte file

by marto (Bishop)
on Oct 13, 2010 at 08:43 UTC ( #865045=note: print w/replies, xml ) Need Help??

in reply to Re^2: Read Some lines in Tera byte file
in thread Read Some lines in Tera byte file

I posted from my phone, I couldn't realistically try this at the time. It is, as I suppose you're suggesting, pretty darn slow. I'd used Tie::File in the past, but not with files this large.


  • Comment on Re^3: Read Some lines in Tera byte file

Replies are listed 'Best First'.
Re^4: Read Some lines in Tera byte file
by ikegami (Pope) on Oct 13, 2010 at 15:56 UTC

    Tie::File remembers the position of the lines it has encountered. Not problem if you want to read lines 100 to 200.

      ...just be sure not to reference $#array or scalar @array, otherwise you have to wait for the whole file to have been shuffled through memory.

      Consuming 1.5GB of ram in the process on my 40e6 line, 3.6GB file. On a Terabyte size file?

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://865045]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (12)
As of 2016-10-27 19:01 GMT
Find Nodes?
    Voting Booth?
    How many different varieties (color, size, etc) of socks do you have in your sock drawer?

    Results (369 votes). Check out past polls.