|Pathologically Eclectic Rubbish Lister|
Slow performance of non-blocking select?by PMCarl (Initiate)
|on Nov 04, 2012 at 21:22 UTC||Need Help??|
PMCarl has asked for the
wisdom of the Perl Monks concerning the following question:
Hello Wise Ones, I'm running into a problem using non-blocking pipes: The select and can_read methods are taking ~300Ás/call (according to NYTProf). Is that normal? It is really killing the performance of my app.
You can see my package at:
I'm new to pipe communication in Perl, so it's very likely that "I'm using it wrong" (yeah, I know, that's what she said)
I'd be happy to take feedback even if it's not related to the performance
I have re-written the Parallel::Fork::BossWorker (http://search.cpan.org/~twilde/Parallel-Fork-BossWorker-0.05/lib/Parallel/Fork/BossWorker.pm) package to support some features that I think are rather critical:
* Messages larger than the pipe buffer, which is 64K on most platforms. The current implementation silently corrupts the data if it's larger than the buffer.
* Asynchronous processing: The current implementation only supports one big batch of processing. I've added a `progress` sub.
Unfortunately, these features require a complete re-write because the current code exploits a feature of pipes where any messages smaller than the buffer are atomic. So the current code can share pipes and doesn't need to manage buffers.
My test is currently taking 50% longer than the original version, which is exactly the amount of time being spent in using select/can_read.