in reply to how to improve the performance of a perl program
After reviewing what you are doing here, and thinking very carefully about it, I am not persuaded that the real problem here will prove to have anything to do with “how many nano-bleems Perl requires to resolve an accessor chain.” Ditto, any and all of the five bullet-points that you have listed here. I seriously doubt that “the root cause of the problem” will actually be found to be meaningfully linked to any of these O(nanoseconds) things. Hence, I doubt that execution-profiling will give you meaningful / useful results.
Instead, I see that you are using a GUI program to “run jobs in thousands,” and that you are using “MySQL to store data.” Anything that you request of MySQL, such that it requires a physical I/O operation to perform, is going to make O(many_milliseconds) ... apiece. Furthermore, you say that you are “using sockets for inter-process communication,” which necessarily means an asynchronous connection, and, I would dare to speculate, likely a home-grown one. Finally, since the reported symptom is “is hung,” as seen from the point-of-view of an event-driven GUI program, I would peg the true culprit, with almost-100% certainty, as “a timing hole.”
I would start troubleshooting this problem by adding timestamps for every significant event to the SQL record that describes each unit-of-work: when was it created, when did it start execution, when did it finish. Then, I would add a very “chatty” event-log file to both the front-end and the back-end systems to record, with timestamps, exactly what each system did and precisely when it did it. I would then run the systems for a little while, find a way to merge the various logs together by time, and look specifically for situations where one piece of the system “didn’t get the message.” Or where, some exception to the expected control-flow actually occurred. I would start with the assumption that the processing sequence isn’t exhibiting piss-poor performance because it is “running out of nanoseconds,” but because it is “getting stuck in traffic” because of a timing-hole that no one had caught ... until now.
If you are barking up the wrong tree, you will never find the raccoon.