Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: how to improve the performance of a perl program

by sundialsvc4 (Abbot)
on Aug 05, 2013 at 18:02 UTC ( #1047942=note: print w/ replies, xml ) Need Help??


in reply to how to improve the performance of a perl program

After reviewing what you are doing here, and thinking very carefully about it, I am not persuaded that the real problem here will prove to have anything to do with “how many nano-bleems Perl requires to resolve an accessor chain.”   Ditto, any and all of the five bullet-points that you have listed here.   I seriously doubt that “the root cause of the problem” will actually be found to be meaningfully linked to any of these O(nanoseconds) things.   Hence, I doubt that execution-profiling will give you meaningful / useful results.

Instead, I see that you are using a GUI program to “run jobs in thousands,” and that you are using “MySQL to store data.”   Anything that you request of MySQL, such that it requires a physical I/O operation to perform, is going to make O(many_milliseconds) ... apiece.   Furthermore, you say that you are “using sockets for inter-process communication,” which necessarily means an asynchronous connection, and, I would dare to speculate, likely a home-grown one.   Finally, since the reported symptom is “is hung,” as seen from the point-of-view of an event-driven GUI program, I would peg the true culprit, with almost-100% certainty, as “a timing hole.”

I would start troubleshooting this problem by adding timestamps for every significant event to the SQL record that describes each unit-of-work:   when was it created, when did it start execution, when did it finish.   Then, I would add a very “chatty” event-log file to both the front-end and the back-end systems to record, with timestamps, exactly what each system did and precisely when it did it.   I would then run the systems for a little while, find a way to merge the various logs together by time, and look specifically for situations where one piece of the system “didn’t get the message.” Or where, some exception to the expected control-flow actually occurred.   I would start with the assumption that the processing sequence isn’t exhibiting piss-poor performance because it is “running out of nanoseconds,” but because it is “getting stuck in traffic” because of a timing-hole that no one had caught ... until now.

If you are barking up the wrong tree, you will never find the raccoon.


Comment on Re: how to improve the performance of a perl program
Re^2: how to improve the performance of a perl program
by BrowserUk (Pope) on Aug 05, 2013 at 19:20 UTC

    You're doing it again. Misreading what is written and making it up as you go along in order to peddle your garbage point of view.

    Finally, since the reported symptom is is hung,

    Nowhere does the OP state that his program is or ever has been "hung".

    He says: "But we are trying to scale up the gui so that it can run nearly 1lac job without getting hung."

    He is looking to prevent; not cure.

    And off the back of that one mismalinterpretation; you MAKE UP an entire scenario -- completely absent from the OPs description -- just in order that you can draw your stupidest conclusion to date. Which given your history of stupid conclusions is a real achievement.

    You sir; are a total moron!


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      My goodness ... bigger type-fonts, too?

      Unforrtunately, the OP really doesn’t give a terrible amount of detail .. perhaps more detail will follow?   Nevertheless, what is said about the program is fairly typical:   that it is a GUI program which is described as “a launching script.”   (Thus my first guess:   that a non-GUI implementation of the same process would behave similarly.   I would not spend any time prowling around the guts of Tk looking for any performance improvement.)   Likewise, the program is further described as “using sockets for IPC” and “using MySQL to store data,” both of which could just-as-equally describe roughly a million other production programs on this planet ... nearly all of which (is this one the exception?) are I/O-bound and prone to have logic-holes in how they handle sockets.   It would be a rare bird indeed if, being such a program, it were throttled by nanoseconds.   (And I know perfectly well that you have written just such “birds.”)   Indeed, one could spend many hours “profiling” such a program, chasing nanoseconds, and although one might come up with interesting results, would never hit upon what made such a program “slow.”   And that is why I suggested that this could be the wrong tree; not some secret desire to be (let alone to be called ...) “a moron” in a public place.

      Do I know?   No.   No more than you do.   Maybe the OP will elaborate further.

        If his central socket server script is to be able to handle 100,000 (1 lakh) concurrent "jobs"; then it will it will be critical to overall throughput that it be responsive to incoming connections/messages. Microseconds will count.

        While the individual jobs may well be IO-bound,if the socket server that sits in the driving seat of all of those processes is unresponsive; then it will have a potentially disastrous affect on overall system throughput by keeping 99,999 jobs waiting --doing nothing -- whilst it deals with the current job.

        All of this is plainly obvious to anyone with a modicum of analytical experience.

        And even to those who bothered actually read what was actually written; rather than imagineering a scenario to fit their personal biases.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1047942]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (4)
As of 2015-07-05 10:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (61 votes), past polls