My scripts routinely process files sized 1GB (gzip compressed). The content is either 28 column CSV (ca. 250 bytes per line) or 500 column fixed width (ca. 2k bytes per record) transaction data. Both types get converted to tab separated output plus two administrative columns and then bulk loaded into database tables, as the bulk loader does not like to talk to a fifo or pipe, unfortunately.
The content of the files is ASCII text, all packed decimals for the fixed width files have already been decoded to numbers.
