Your skill will accomplish what the force of many cannot |
|
PerlMonks |
Re^3: Super fast file creation neededby thezip (Vicar) |
on Oct 19, 2007 at 06:26 UTC ( [id://645904]=note: print w/replies, xml ) | Need Help?? |
I understood that the logfiles were huge, which IMHO, makes storing the entire lines as hash keys impractical due to memory considerations. Sure, computing checksums/digests might slow things down some, but it is one way to identify whether a line has been seen or not. With the proper digest length, hash key collisions could be virtually eliminated. In this case, I think the memory considerations outweigh the speed considerations, but it would certainly be prudent to benchmark both ways to see which one works better. Where do you want *them* to go today?
In Section
Seekers of Perl Wisdom
|
|