Re^2: processing huge files

[jhourcle]

If you're going to hand off the work to dd, you might want to use split, as it can act on full lines (so won't break in the middle of a record, given the logic the OP was using.).

You also don't need to recursively call it, as the equivalent to your dd example would be:

split -b 30720m -a 3 $INFILE

[fauria]

I used dd because it can access directly a position in a file using skip, and then sequentialy read its content, without needing to load the whole file and then point to a location.