Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re^3: Read Some lines in Tera byte file

by marto (Bishop)
on Oct 13, 2010 at 08:43 UTC ( #865045=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Read Some lines in Tera byte file
in thread Read Some lines in Tera byte file

I posted from my phone, I couldn't realistically try this at the time. It is, as I suppose you're suggesting, pretty darn slow. I'd used Tie::File in the past, but not with files this large.

Cheers


Comment on Re^3: Read Some lines in Tera byte file
Replies are listed 'Best First'.
Re^4: Read Some lines in Tera byte file
by ikegami (Pope) on Oct 13, 2010 at 15:56 UTC

    Tie::File remembers the position of the lines it has encountered. Not problem if you want to read lines 100 to 200.

      ...just be sure not to reference $#array or scalar @array, otherwise you have to wait for the whole file to have been shuffled through memory.

      Consuming 1.5GB of ram in the process on my 40e6 line, 3.6GB file. On a Terabyte size file?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://865045]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (8)
As of 2015-07-30 11:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (271 votes), past polls