Optimization Help on Perl Hash Traversal (eval use)by mwb613 (Sexton)
|on Feb 06, 2013 at 21:56 UTC||Need Help??|
mwb613 has asked for the
wisdom of the Perl Monks concerning the following question:
Thanks for looking.
I am hoping to optimize (read: speed up) a script that produces statistics from call log files. The files are flat files in which each line describes a leg of a SIP phone call. Each call has a unique ID but the call could have multiple legs (therefore multiple occurrences of the ID in each file. A short summary of the method would be this:
1: Loop through log file and create a hashref keyed on the unique ID of the call that captures certain columns from the file. e.g.:
2: Here's where the slowness is coming in. I loop through the resulting hash (based on session_id) and use an inner loop to run a few eval blocks in order to calculate my statistics. The strings that I put in the eval block are pulled from a hashref I created from a "config table" in mySQL (uses a fetchall_hashref so I'm only doing it once).
3. Loop through newly created hash and push stats to a mySQL database. This is working very quickly.
The idea behind the eval blocks is to add a layer of abstraction so that when I need to add additional statistical analyses I can add entries to mysql with the proper eval string. I know this could be done via flat file or XML but I don't believe it is costing any extra time to dip the DB once to get my eval strings.
I am looking to have as near to real time statistics as I can but the above process is really bogging down during the loop which groups data into the second hashref. For my purposes, which is to push this data into a database that can be queried and graphed, the process will get bogged down to the point where I don't believe I'll be able to catch up. (The logs cut off every 5 minutes) and the process is taking about that long on a large file (100k rows). When I look at TOP it shows Perl using about a full processor but not a whole lot of memory (~4%).
I'm hoping that someone might have an idea that could help speed the process up. I'm not looking for anyone to write code for me. Just point me in the right directions or drop a few cryptic terms that I can research.
This is done in Perl 5.10.1 on Centos 6, fyi.