note hippo <p>If by "1B" you mean 10^9 and if your fields have mean length 9 chars, then including tabs you have roughly 500GB in one file, correct? I'm not surprised it is very slow. How fast to just cat the file? How much slower is your script?</p> <p>Best advice is buy the fastest disk you can afford. And maybe think about preprocessing.</p> 1014517 1014517