Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Re^2: Read Some lines in Tera byte file

by BrowserUk (Pope)
on Oct 13, 2010 at 06:55 UTC ( #865023=note: print w/replies, xml ) Need Help??


in reply to Re: Read Some lines in Tera byte file
in thread Read Some lines in Tera byte file

use something like Tie::File,

Please try this yourself--say read the middle line in a 1 or 2GB file--before suggesting it to others.

  • Comment on Re^2: Read Some lines in Tera byte file

Replies are listed 'Best First'.
Re^3: Read Some lines in Tera byte file
by marto (Archbishop) on Oct 13, 2010 at 08:43 UTC

    I posted from my phone, I couldn't realistically try this at the time. It is, as I suppose you're suggesting, pretty darn slow. I'd used Tie::File in the past, but not with files this large.

    Cheers

      Tie::File remembers the position of the lines it has encountered. Not problem if you want to read lines 100 to 200.

        ...just be sure not to reference $#array or scalar @array, otherwise you have to wait for the whole file to have been shuffled through memory.

        Consuming 1.5GB of ram in the process on my 40e6 line, 3.6GB file. On a Terabyte size file?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://865023]
help
Chatterbox?
[liz]: ^10 is just a short way of saying 0..^10, which is a short way of saying 0..9
[liz]: ^N is a very common idiom in Perl 6 to indicate "N times"
[liz]: The ^ indicates "exclude endpoint from range"

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (8)
As of 2018-07-16 16:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    It has been suggested to rename Perl 6 in order to boost its marketing potential. Which name would you prefer?















    Results (344 votes). Check out past polls.

    Notices?