Clients glomming onto an SSL listener

[mungohill]

Greetings Esteemed Monks

I haven't been near the monastery for a couple of years because either stuff has been working or I can figure out what is wrong with my feeble powers, but this one's got me beat.

As I seem to have stunned the monks into silence with this one, I've also posted a similar query to [e-mail], especially as it does seem more of an openssl control problem. I'll let you know if I get anything from that direction

I've had some useful response from the SSL community along the lines that I should expect this problem if I have a single-threaded listener. Also some very sensible recommendations about what to do. Basically it's a really bad idea to use the accept method of IO::Socket::SSL as, unless you have really trivial traffic levels, you will get SSL handshake collisions that completely take your service out. Better to do a basic tcp accept on the socket, and on return, fork, (and in my case do the tcpwrap checks) then invoke start_SSL, passing it the tcp socket which gets blessed to an IO::Socket::SSL socket if the handshake succeeds. All handshake is then done in separate processes. Seems to work so far.

I have a service that presents a very simple SSL listener which, when it is hit, identifies the connecting node from its certificate/peer address and just splurges some xml at them containing data from some files in the queue directory that contains their data.

All the client does is to open a socket and start reading.

This works, but it is unpleasantly susceptible to problems which I believe are caused by clients with bad internet connections (the pathology suggests this). It seems that something unpleasant occurs in the SSL handshake process which causes the socket to hang indefinitely. Nobody else can connect when this has happened.

I currently have a very crude monitor program that checks the activity log of the main service and, if things go suspiciously quiet it bludgeons it into compliance by restarting the listener service. I also logs the result of a filtered run of 'netstat' which shows who was involved with the problem. Some hosts feature a lot more regularly, which leads me to believe is it some net environment problem.

But that's all a distraction. The fact is that the service shouldn't be susceptible to the vagaries of what stupid clients or bad networks do.

I *do* get quite a lot of action on my SSL_error_trap routine. These errors log nicely and don't cause a hang and don't seem to coincide with any hangs.

Pretty much immediately after the accept the program forks a handler, but the rogue clients must be glomming onto the main process before the SSL negotiation is complete.

I can't help thinking that I should be able to tell SSL to have some sensible (fairly aggressive) timeouts on connections that fail an SSL handshake. Is this possible? Does it sound like I'm even identifying the problem correctly?

--I'm wondering if I might get more control using non-blocking but I can't figure out if that's just a Windows thing

Perl version v5.8.8 program has:

use IO::Socket::SSL;
use IO::Socket::INET;
use parse_config;
use makemess;
use POSIX ":sys_wait_h";
use Net::TCPwrappers qw(RQ_CLIENT_ADDR RQ_DAEMON RQ_FILE request_init 
+fromhost hosts_access);
use capserr;
use Cwd;
use strict;
use warnings;
.
.
.
 if(!($sock = IO::Socket::SSL->new( Listen => 5,
                   LocalAddr => $setup{MYaddr},
                   LocalPort => $setup{CAPSslportS},
                   Proto     => 'tcp',
                                   SSL_verify_mode => 0x01,
                                   SSL_cert_file => 'certs/capscert.pe
+m',
                                   SSL_key_file  => 'certs/capspk.pem'
+, 
                                   SSL_ca_file => 'certs/cacert.pem',
                                   SSL_error_trap => \&SSL_FAIL,
                   Reuse     => 1,
                 )) ) {
       print "unable to create socket: ", &IO::Socket::SSL::errstr, "\
+n";
       exit(0);
   }

[flexvault]

mungohill

You may want to take a look at PROCESS: Too many child process, serving the connection. Symptoms sound similar.
Good luck!

[mungohill]

I think I've got something different. There's no evidence of a succeeded accept followed by a failed fork (which as I understand it is the essence of what was being described). I just watched an 'incident' just now, and for every succeeded accept, there was a successful fork. My hangup monitor showed an earlier hangup like this:

2011-05-10 14:39:52|No systems connected to listening sender since 201
+1-05-10 14:29:31

tcp        0   1413 xxx.xxx.143.138:6023        yyy.yyy.1.172:redstorm
+_join  ESTABLISHED 29021/perl          
2011-05-10 14:39:52|Restarting daemons
root     29021     1  0 13:51 pts/2    00:00:05 /usr/bin/perl /usr/sbi
+n/caps/lsenders.pl -d
root     29029 29021  0 13:51 pts/2    00:00:00 /usr/bin/perl /usr/sbi
+n/caps/lsenders.pl -d

this shows the output of a call to netstat showing that a connection from a particular node was attached to the 'main' listener process (not a forked 'servicer'). The situation is rather confused by the fact that there is a permanently forked-off 'monitor' process (2029) that (I hope) isn't involved in any way in this problem. When this happens, the 'stuck' node is visibly 'stuck' to the listener indefinitely