in reply to Dealing with huge log files 100 MB to 1 GB
This isn't useful for statistics, but if you need to quickly pull up sections of the large files to diagnose things like "what happened around time x?", File::SortedSeek is incredibly useful.
Mike
For your current problem, as everyone else said "precompute" is probably the best answer.
You could probably do something complicated to map/reduce the statistics gathering in parallel across sections of the file, but it's probably not worth it.
Mike
|
|---|
In Section
Seekers of Perl Wisdom