Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Large HTTPS POST hangs with HTTP::Daemon::SSL

by dgaramond2 (Monk)
on May 01, 2009 at 12:27 UTC ( #761270=perlquestion: print w/replies, xml ) Need Help??

dgaramond2 has asked for the wisdom of the Perl Monks concerning the following question:

I'm not sure yet if this is a bug with the module, so I'm posting the problem here first to confirm whether others are experiencing the same, and to gain some insights.

I'm writing an HTTPS daemon using HTTP::Daemon::SSL. It runs fine except when receiving POST requests with content larger than around 66-67k. The get_request() method won't return, as if it were trying to read more data, until the HTTP client breaks the connection and then get_request() will return the complete HTTP::Request object normally.

* GET and smaller POST requests are ok.

* The problem doesn't happen if I use HTTP instead (HTTP::Daemon).

* For the client I have tried LWP::UserAgent and wget, with the same result.

* I have tried localhost ( as well as public address, with the same result.

* I have also tried using HTTP1/.0 instead of HTTP/1.1, with same result. To do this with LWP, you set env PERL_LWP_USE_HTTP_10=1. I'm not sure how to do this with wget though.

For those who want to reproduce the problem, here's how:

#!/usr/bin/perl -w
use strict;
use HTTP::Daemon::SSL;
use HTTP::Response;

my $server = HTTP::Daemon::SSL->new(
    LocalPort => 8010,
    Reuse => 1,
    Timeout => 180,
    SSL_key_file  => "server-key.pem",
    SSL_cert_file => "server.crt",

print "Server started on port 8010\n";
while (my $c = $server->accept) {
    print "Connection from ".$c->peerhost."\n";
    my $req = $c->get_request or next;
    print "Request dump: ".$req->as_string;
    my $resp = HTTP::Response->new(200);

Before running, create server-key.pem and server.crt:

$ openssl genrsa -out server-key.pem 1024
$ openssl req -new -key server-key.pem -out server.csr
$ openssl req -x509 -key server-key.pem -in server.csr -out server.crt

Ignore any questions about country code, etc and just press ENTER.

Then run, and then try to connect using wget with some data:

$ dd if=/dev/urandom of=post66k bs=1k count=66
$ dd if=/dev/urandom of=post67k bs=1k count=67
$ dd if=/dev/urandom of=post500k bs=1k count=500
$ wget -S -O- --no-check-certificate; # normal
$ wget -S -O- --no-check-certificate --post-data 12345; # normal
$ wget -S -O- --no-check-certificate --post-file post66k; # normal
$ wget -S -O- --no-check-certificate --post-file post67k; # hangs
$ wget -S -O- --no-check-certificate --post-file post500k; # hangs

For the tests that hang, try pressing Ctrl-C to exit wget and the server will return the completed request normally.

Modules used are the most recent as of this writing: IO::Socket::SSL 1.24, HTTP::Daemon::SSL 1.04, LWP 5.826.

Tried on Debian Lenny on i686, which includes Perl 5.10.0. I have also tried this on Perl 5.8.8 on Debian Etch with same results.

  • Comment on Large HTTPS POST hangs with HTTP::Daemon::SSL

Replies are listed 'Best First'.
Re: Large HTTPS POST hangs with HTTP::Daemon::SSL
by derby (Abbot) on May 01, 2009 at 14:08 UTC

    Hmmm ... well this is just odd and I do don't know the full reason but it appears to be blocking on the select in HTTP::Daemon::ClientConn::_need_more method. Removing the Timeout from the HTTP::Daemon::SSL constructor enables you to avoid the select here and the large requests pass through fine.

Re: Large HTTPS POST hangs with HTTP::Daemon::SSL
by zwon (Abbot) on May 01, 2009 at 14:32 UTC

    I've just reproduced the problem using 37k of data. It looks like select-sysread loop in HTTP::Daemon doesn't work correctly with IO::Socket::SSL. That's because select in _need_more tests real filehandle and sysread reads from IO::Socket::SSL object which is buffered, so

    sysread($self, $_[0], 2048, length($_[0]))
    may actually read more than 2048 bytes from the socket and subsequent select on socket will hang.

    Removing Timeout as proposed by derby solves the problem.

Re: Large HTTPS POST hangs with HTTP::Daemon::SSL
by dgaramond2 (Monk) on May 01, 2009 at 17:00 UTC
    Thanks guys. For the moment I'm removing the Timeout options in the constructor. I'll also try to submit a bug report to RT.
Re: Large HTTPS POST hangs with HTTP::Daemon::SSL
by aufflick (Deacon) on May 03, 2009 at 08:43 UTC
    Hi guys,

    I'm the current maintainer of HTTP::Daemon::SSL which I took over from Peter Behroozi to apply some patches back in 2007.

    I see the problem, and I can't think of any smart alternatives at the moment. I don't like the idea of HTTP::Daemon::SSL not supporting timeouts since it is then not a full drop in replacement for HTTP::Daemon, but it's obviously better than a big bug such as this.

    So unless I (or others on this thread) have any genius ideas in the near future, I'll:

    1. add a test for the large post issue;
    2. add a TODO test for the timeout feature; and
    3. disable the timeout feature.

    Better ideas welcome!

Re: Large HTTPS POST hangs with HTTP::Daemon::SSL
by dirack (Initiate) on May 05, 2009 at 15:26 UTC
    I suggest overriding select function in module HTTP::Daemon::ClientConn.
    The new select function checks args & if "our case" happens it calls IO::Socket::SSL::peek function instead of CORE::select.
    I have tested the code on my server - it's works fine. Timeout in this implementation should work too. plz have a look at the patch:
    diff -u orig/ mod/
    --- orig/	2008-02-12 03:27:03.000000000 +0200
    +++ mod/	2009-05-05 17:38:50.000000000 +0300
    @@ -69,6 +69,11 @@
     use vars qw($VERSION @ISA $PROTO $DEBUG);
    +BEGIN {
    +no strict;
    +*{"HTTP::Daemon::ClientConn::select"} = \&HTTP::Daemon::SSL::select;
     use IO::Socket::SSL;
     use HTTP::Daemon;
    @@ -93,6 +98,42 @@
    +my %save_handlers;
    +sub select($$$$)
    +    my ($r,$w,$e,$timeout) = @_;
    +    unless (!defined($w) && !defined($e)) {
    +    return CORE::select($r,$w,$e,$timeout);
    +    } 
    +    # may be our case.
    +    # detail check this:
    +    my $value = vec($r,0,length($r)*8);
    +    return CORE::select($r,$w,$e,$timeout) if ( ($value & ($value - 1)) or (!$value)) ; # must only one bit set (see HTTP::Daemon code)
    +    my $fd;
    +    for($fd=0;$fd < length($r)*8;$fd++) { last if (vec($r,$fd,1));};
    +    die "bad thing happen ;-(" if ($fd == length($r)*8);
    +    return CORE::select($r,$w,$e,$timeout) unless (defined($save_handlers{$fd}));
    +    # ok. it's our case
    +    my ($buf,$res);
    +    { # set timeout of operation
    +    local $SIG{ALRM} = sub { die "alarm\n" };
    +    alarm $timeout;
    +    $res = $save_handlers{$fd}->peek($buf,1);
    +    alarm 0;
    +    }
    +    if (!defined($res) or $res < 0) {
    +    # error or timeout
    +    # FIXME: timeout ???
    +    return undef;
    +    }
    +    # $res >= 0) 
    +    return 1;
     sub new
         my ($class, %args) = @_;
    @@ -109,6 +150,7 @@
     	my ($sock, $peer) = IO::Socket::SSL::accept($self,$pkg);
         if ($sock) {
             ${*$sock}{'httpd_daemon'} = $self;
    +        $save_handlers{fileno($sock)} = $sock;
             return wantarray ? ($sock, $peer) : $sock;
         else {
      I include HTTP::Daemon::SSL like that:
      use HTTP::Daemon::SSL;.
      But if I wrote:
      use HTTP::Daemon;
      use HTTP::Daemon::SSL;

      that code unfortunately fail.
      It happent because code:
      BEGIN {
      no strict;
      *{"HTTP::Daemon::ClientConn::select"} = \&HTTP::Daemon::SSL::select;
      must execute before use HTTP::Daemon.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://761270]
Approved by ww
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (5)
As of 2021-06-14 13:03 GMT
Find Nodes?
    Voting Booth?
    What does the "s" stand for in "perls"? (Whence perls)

    Results (62 votes). Check out past polls.