|
|
|
| Problems? Is your data what you think it is? | |
| PerlMonks |
Re: Huge files manipulationby educated_foo (Vicar) |
| on Nov 10, 2008 at 12:09 UTC ( [id://722637]=note: print w/replies, xml ) | Need Help?? |
|
You need to do some sort of divide-and-conquer approach. For example, you could split up the data into a bunch of temporary files according to the first 2-3 fields, uniquify each of those files, then cat them back together into one big file.
In Section
Seekers of Perl Wisdom
|
|
||||||||||||||||||||||||||||||