in reply to Re^2: Multithreaded process on AIX slow
in thread Multithreaded process on AIX slow
Devel::NYTProf is not currently thread safe or multiplicity safe.
Sorry. I don't use NYTProf myself, so I've never encountered the limitation. I only mentioned it because it seems to be what everyone else recommends.
This application cannot be run - or rewritten to run - without threads.
There would be no point to doing so as the bottlenecks would undoubtedly move.
But, the core of my advise above remains. You will need to profile.
I tend to do this manually. I find that profiling every line takes too long and produces far too much output to be useful.
So, I start course-grained at first till I find the points of interest, then more and more fine-grained around those points of interest until I find what I'm looking for.
So, for a queue consumer thread I will just trace:
printf STDERR "[%3d] %u %s:%d %s\n", threads->tid, time(), __FILE__, _ +_LINE__, $workitem;
once for each loop of the queue processing loop.
That should have negligible affect on overall performance of the code, but should give sufficient information to see if one or more your threads and/or particular work items are significantly slower on the AIX machine that the others.
Once you get some indication of where the differences lie, you can then disable or remove the irrelevant traces and add a few more in the appropriate places to allow you to zoom in on the cause(s).
One possibility -- based upon something you said in one of your other replies above --
are you using virtualisation on one or both of the systems? If so, how are you configuring the vm that is running this app (Ie. what cores/cpus/virtual network adapters/etc)?Because historically, if you configure a VM with equal or more of any of those than the actual machine it runs on has, then it can lead to some very bad interactions between the host and guest OS schedulers and lead to diabolically bad performance. Just a thought.