|No such thing as a small change|
Multithreaded process on AIX slowby scunacc (Acolyte)
|on Nov 13, 2013 at 19:55 UTC||Need Help??|
scunacc has asked for the
wisdom of the Perl Monks concerning the following question:
UPDATE: Interesting. The REST calls I make require authentication. I get an authentication token and then make a 2nd call. I decided to cache the authentication token per thread (which worked - wasn't sure if it would for multiple successive calls) and it dropped *2 hrs* off the AIX run time. So, making the REST calls from AIX is definitely a source of the problem. Not sure why yet. Interestingly, doing the same on the Xeon Intel machine running Linux dropped the runtime to 52 mins from 1.5 hrs. So, not so much of an improvement but definitely better there too.
ORIGINAL QUESTION BELOW
Skip to the bottom for the Q if you don't want the background here:
I have a mature robust multithreaded application that's been running in various guises and various UN*X flavors since 2008. I have just modified it to add some new features and run in a different way. The process can spawn up to several hundred threads at a time which remain running to multiplex and queue multiple parallel client inputs on multiprocessor machines.
In the new version, I hit an external website with a REST query rapidly - multiple times per second with several hundred clients talking to the server process threads sequentially - 20 at a time.
When I run this on an old Pentium D Dual core machine, it takes 2 hrs to run thru' all my clients. On a Xeon quad core, it's an hour and a half. The network response times do dominate a bit - I'm already aware of that. However, on AIX, the thing takes **4 hours** to complete! That.Should.Not.Happen! :-) The AIX machine is a multiprocessor workhorse with partitioning and 64G memory. It beats the pants off of the Xeon-based machine, and certainly the Pentium D machine.
So, - my Q is, why would this take so long on AIX? Is it a function of using threads on AIX? Something in the REST client libs? (Using REST::Client) something else odd that anyone is aware of on AIX? Remember - I've been using this software in one form or another on AIX for years (I wrote it - it's about 60k lines of code overall). Only noticed this problem because I had a comparison baseline with other systems on this particular instance of the server, and other uses of it didn't hit the problem. I'm thinking something in the net calls or something in the context switching among threads.
Corollary: I have a way to batch the REST query data. When I do that, on the Pentium D, the data set takes 18 mins (but is less accurate). When I batch it on AIX it takes *15* mins. Yes - right - so - something to do with networking or context switching. I'm pitching to networking.
Corollary 2: The Xeon machine is on the same internal network as the AIX machine. So, not a network connectivity problem per se.
Fellow Perl folks. Your help - as always - is much appreciated.
Kind regards, Derek