Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re^3: Split a file based on column

by space_monk (Chaplain)
on Jan 17, 2013 at 11:04 UTC ( #1013758=note: print w/replies, xml ) Need Help??


in reply to Re^2: Split a file based on column
in thread Split a file based on column

You caught my comment whilst it was being drafted; I did state another reason for the approach I suggested.

Memory is almost never a problem nowadays unless you're running it on your 15 year old PC, but 300k rows * 64 k per row (19GB??) may give some pause for thought. Time to go shopping for more memory or increase your cache. :-)

A Monk aims to give answers to those who have none, and to learn from those who know more.

Replies are listed 'Best First'.
Re^4: Split a file based on column
by davido (Archbishop) on Jan 17, 2013 at 18:56 UTC

    Loading a 19GB file into memory does indeed give pause for thought.... long long pause. :) Time enough to contemplate approaches that do scale well.

    Your accumulate and write when full strategy is a pretty good idea. It would be a data cache rather than a filehandle cache, and the implementation ought to be pretty straight forward. Implementing the file-handle LFU cache seems like it would be more fun though.


    Dave

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1013758]
help
Chatterbox?
[Corion]: Meh, I found another Amazon "used" scam reseller... If a product sells for "almost new" at half the price, it's a scam, most likely...
[ambrus]: Corion: that's not true. Actually for Christmas and Thanksgiving, a lot of people buy electronics such as cameras as present, then many of them figure out they don't need it,
[ambrus]: and the electronics gets reselled almost new, but it has to be sold at half price because otherwise everyone chooses to buy the new product which has fewer risk of selling damaged products labelled as almost new.

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (15)
As of 2017-11-21 14:57 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In order to be able to say "I know Perl", you must have:













    Results (304 votes). Check out past polls.

    Notices?