in reply to Alternate for "open"
Are your data files binary or text? Can you show us a small sample input file?
Update: I see by your code now that the files are text. Is it possible that they have such long lines that the read via the diamond oprator is taking a huge chunk of memory?
Update 2: Duh, I'm still asleep--please forgive my incoherent rambling about things I failed to notice in the OP.