|Perl: the Markov chain saw|
Re: Poor performances with threadsby BrowserUk (Pope)
|on Oct 23, 2009 at 11:18 UTC||Need Help??|
It doesn't help that you are making so many copies of your data and results:
And three of those copies are as lists onto the stack, which then get copied again once and often twice into data structures internal to Statistics::Basic.
I suspect, but cannot verify without trying to re-create your script, (why not post the whole thing if you want help?), that you are spending most of your time allocating memory and copying data, rather than calculating.
Here is a simple script that calculates the coefficient of variation of 55k 7-element datasets in just over half a second:
Re-using other peoples code is all well and good, but when it leads to you writing and supporting far more complex code yourself in order to meet your operational performance requirements, because you are constantly having to restructure the natural state of your data to force-fit it to those of the modules you use--and you have no choice but to inherit a bunch of functionality, or just prepreparation for that functionality that you do not require--then it is time to consider extracting just that code that you actually need from the module and make your life simpler.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.