Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Working with a very large log file (parsing data out)

by generator (Monk)
on Feb 21, 2013 at 00:32 UTC ( #1019871=note: print w/ replies, xml ) Need Help??


in reply to Working with a very large log file (parsing data out)

I'd build a hash using the log entry date as the key and the single field value as the (presumably numeric) value. As each line in the source log file is read, test for the existence of the key, if found increment the value by the current line's field value. If it is not found create a new key value pair in the hash. Sorting the hash after completing the processing of the file should be significantly less memory intensive as you'll be sorting the summary records not the detail records. That's my 2 cents for what it's worth.

<><

generator


Comment on Re: Working with a very large log file (parsing data out)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1019871]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (7)
As of 2014-09-20 18:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (160 votes), past polls