I have another suggestion, which may or not be appropriate. In the work I'm currently doing, I have to apply a whole pile of rules to input and output records. For each decision that I make (this policy is the wrong type), I put a standardized message out to a log file. Afterwards, if I want to follow up on a particular policy, I grep for just that policy number, and I can see everything that happened to it.
I also have the ability to run the script in a "Just process this policy" mode, rather than do everything. It still has to read through all of the input files, but it means that the data structures only have information regarding that policy, so a dump of the data structures is quick and complete. Perhaps that would be handy for your script -- specify just a single file, in order to investigate (using the debugger) exactly what's going on.
Alex / talexb / Toronto
Thanks PJ. We owe you so much. Groklaw -- RIP -- 2003 to 2013.