How could it be hardware related if these files are on
disk at that size and C++ programs process them?
how do you process your files with perl? Is it a specfic perl? or compile option? or algorithm?
Easily! However, you never said that you had other programs that were successfully processing your data. =)
As far as my processing is concerned, I churn through Cisco NetFlow data, along with bgp tables, route summaries, snmp data, merge it all together and spit it back out again to create traffice matrices. The algorithms aren't that complicated at all; it's just a shite-load of data that I process continuously. If I tried to run on anything less than the E450 that it runs on, I would be in trouble.