Socket IO with large (>1000) numbers of open sockets

[Ray Smith]

Folks, I'm looking for suggestions on how I might improve the efficiency of a program I use which does non-blocking HTTP io with often 1000+ open sockets. The central action of the program is characterized by the simplified code snipit which follows later. My thanks to liverpole for reminding me of the Perl module IO::Select which, although I had previously used, did not in this code which I inherited from the original code author. I suspect that much time is taken by checking on socket availability too often. I hope that there is a method to limit my calls to IO::Select:can_read so they are only done only when there is pending IO. I have been unsuccessful in finding such a mechanism. Is there any mechanism to implement the following pseudocode more efficiently than just calling IO::Select's can_read() every time one needs to check if any socket io is pending?

$SIG{INTERUPT_ON_PENDING_SOCKET_IO} = \&ckSockets;

Another optimization possibility

Even though we may have 1000+ open sockets the activity at any one time is sparse. I've been speculating about going back to the bit vector version of select and looking at the 1000+ bit length vector 32 at a time. I'm not optimistic about this approach. For all I know, the implementer of IO::Select may already do this. Highly simplified version of my current code

use IO::Select;
...
# Check for and process any pending socket input
# avoid steping on toes by keeping running list of ready
# sockets and process it untill empty
sub ckSockets {           # Returns: # of ready so we can tell activit
+y
    ...
    my @breadys = $io_select_obj->can_read(0);
    foreach my $fd_key (@breadys) {
        ...read and process data from this socket...
}

Looking at the top of profiled run below we see that the time appears dominated by calls to the socket testing:

[root@ibm-blade-blade0 testbuddy]# time dprofpp
Total Elapsed Time = 1790.060 Seconds
  User+System Time = 1315.990 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 55.9   736.8 753.00 317833   0.0002 0.0002  IO::Select::can_read
 9.05   119.1 256.27 363021   0.0000 0.0001  BuddyUsers::log
 5.25   69.14 137.12 350999   0.0000 0.0000  tsprint::ts
 5.17   67.97 67.979 350999   0.0000 0.0000  POSIX::strftime

Update:  Updated to correct spelling on IO::Select, add '0' to can_read to reflect actual code.

[Util]

Investigate IO::Poll. If it works on your platform (non-BSD Unix?), it *may* provide much better performance with a large number of open sockets.

Caveat: I have not tried this myself. In fact, there is a decent chance I am *completely* mis-remembering select/poll distinctions.

[BrowserUk]

How much cpu does your server use when it's polling those 1000 connections?

I ask, because think that you may be misinterpreting the results of your profiling.

What the profiling shows is that your program is spending 56% of elapsed wall-clock time in the can_read() call, but so what? You are not using a timeout value, which means that if there is nothing to do, no input to read, then that call waits until there is. What else would you have it do?

Unless you are thrashing your cpu, that is probably exactly where you want it to wait whilst there is nothing to do so that it is instantly read to roll as soon as there is.

If you are thrashing the cpu that is a different matter, but that is not reflected or demonstrated in the profiler output.

In that case, it may be that the implementation of IO::Select or the underlying calls on your system is such that whilst there is nothing avialable to read, it sits in a tight loop--like cartoon kids on a road trip:Are we there yet? Are we there yet? Are we there yet?--, and that is the cause of the thrashing.

In that case, it might be better to add a timeout to the can_read() call and explicitly sleep a little if there is nothing to do.

sub ckSockets {
... CHECK:
    my @breadys = $io_select_obj->can_read( 0.01 );
    unless( @breadys ) {
        usleep( 0.1 ); ## See Time::HiRes
        ## Or select undef, undef, undef, 0.1; 
        ## but that might be a "busy loop" also.
        goto CHECK;
    }
...
}

But, unless you are experiencing high cpu loads when there is little or no traffic, then can_read() is exactly the right place for you program to spend most of it's time whilst it waits for something to do. Good on you for profiling, but be sure you are interpreting the results correctly.

[Ray Smith]

Thanks for the observations. I must appologize for the typo of "can_read()" introduced by my attempt to summarize the actual code. I really use can_read(0) because I do lots of work when there is no input to process. You did help me find a bug in my code though. The new profile times appear close. Unless I'm missing something here, I seams that the time is taken by checkintg to see if there are ready sockets and gathering those sockets.

  User+System Time = 1255.587 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 56.9   714.4 733.43 341511   0.0002 0.0002  IO::Select::can_read
 4.47   56.09 87.694 364493   0.0000 0.0000  BuddyUser::log
 3.96   49.68 1294.3 337516   0.0000 0.0004  BuddyUser::ckUser
 3.13   39.35 772.78 341511   0.0000 0.0002  HttpNb::ios_can_read

[BrowserUk]

I seams that the time is taken by checkintg to see if there are ready sockets...

There is no way to know from the information you are supplying. Ie. a profile; and either entirely non-representative code or no code at all; and no indication of how much cpu your application is typically using.

You say that you are doing lots of "other stuff" when no input is available, but what if there is none of that 'other stuff' to do either?

In that case, you'll check for input and find nothing and wait no time. You'll check for 'other stuff' and find nothing to do, so you'll go back and check for input again and find nothing, so you'll check for...

It's a tight loop with most of the work (polling 1000 sockets) occuring inside can_read(). So that what profiling shows you.

Again, unless you are using an abnormal amount of cpu when there is little or no socket traffic, the profiles you are posting are not necessarily indicative of a problem.

If you are using a large amount of cpu when there is little or no socket traffic, then it would probably benefit your application to insert a small sleep somewhere.

Eg. if( @ready = can_read( .01 ) ){. That would reduce the 'busy loop' effect when there is nothing to do, and so reduce your cpu usage, but it would probably increase the proportion of time spent in can_read().

But really, without more information, it's just speculation.

[dmitri]

Have you ever tried Event::Lib? It's built on top of libevent, which makes non-blocking network programming almost embarrassingly simple.