http://www.perlmonks.org?node_id=1014519


in reply to selecting columns from a tab-separated-values file

If by "1B" you mean 10^9 and if your fields have mean length 9 chars, then including tabs you have roughly 500GB in one file, correct? I'm not surprised it is very slow. How fast to just cat the file? How much slower is your script?

Best advice is buy the fastest disk you can afford. And maybe think about preprocessing.