Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Re^2: Get Timout with LWP

by IlyaM (Parson)
on Jun 21, 2004 at 15:54 UTC ( #368490=note: print w/replies, xml ) Need Help??

in reply to Re: Get Timout with LWP
in thread Get Timout with LWP

LWP::Simple is just that. Simple. It will hang if it can open a socket to a server but the server then plays dead and does not respond. LWP::UserAgent is much more sophisticated with its timeouts and will return if it times out.......

This is quite wrong. LWP::Simple is just a wrapper on top of LWP::UserAgent so it behaves exactly same way as LWP::UserAgent concerning timeout handling. The only exception is that LWP::Simple always uses default timeout value (180 seconds) and with LWP::UserAgent timeout can be set.

Ilya Martynov,
Quality Perl Programming and Unix Support UK managed @ offshore prices -
Personal website -

Replies are listed 'Best First'.
Re^3: Get Timout with LWP
by tachyon (Chancellor) on Jun 22, 2004 at 01:45 UTC

    Unfortunately I am not wrong. The hanging behaviour is real. The underlying reason is LWP::Simple uses (for http) the _http_trivial_get() function, not LWP::UserAgent. This implements a 60 second IO::Socket::INET make connection timeout, but no timeout on data recieve. Modifying the test code to use LWP::Simple proves the point (LWP::Simple will wait 500 seconds to get the data - it gets an instant socket, then nothing for 500 seconds). This function remains active in the latest LWP::Simple.

    I agree it should really be a subclass of LWP::UserAgent but for http it is not. The timeout behaviour for LWP::Simple and LWP::UserAgent could IMHO use some work.

    C:\>type #!/usr/bin/perl use IO::Socket; use IO::Select; $lsn = IO::Socket::INET->new( Listen => 1, LocalAddr => 'localhost', LocalPort => 9000,); my $client = new IO::Select( $lsn ); while( my @ready = $client->can_read ) { for my $fh (@ready) { if($fh == $lsn) { warn "Accepted new socket\n"; my $new = $lsn->accept; $client->add($new); } else { # Process socket warn "Getting data\n"; $data = <$fh>; # yeah yeah warn "Got $data\nDoing nothing forever!\n"; my @response = split '', "HTTP/1.1 200 OK\n\nHello World!\ +n"; sleep 500; print {$fh} @response; $client->remove($fh); $fh->close(); } } } C:\>type #!/usr/bin/perl $|++; use LWP::Simple; $start = time(); print "Begin at $start\n"; $response = get('http://localhost:9000/'); $end = time(); my $time = $end - $start; print "Done at $end\nTook $time seconds\nGot:\n$response\n"; C:\>perl Begin at 1087868116 Done at 1087868616 Took 500 seconds Got: Hello World! C:\>



Re^3: Get Timout with LWP
by tachyon (Chancellor) on Jun 22, 2004 at 02:47 UTC

    Further to the previous, here is a possible patch to implement timeout behaviour in LWP::Simple. There is no way to set $timeout passed to _trivial_http_get() included. A global would be easiest I guess given the very simple interface. Defaults to 180 seconds.

    $ diff -u Perl/site/lib/LWP/ Perl/site/lib/LWP/Simple-patch.p +m --- Perl/site/lib/LWP/ Tue Jun 22 12:40:14 2004 +++ Perl/site/lib/LWP/ Tue Jun 22 12:42:15 2004 @@ -154,9 +154,10 @@ sub _trivial_http_get { - my($host, $port, $path) = @_; + my($host, $port, $path, $timeout) = @_; #print "HOST=$host, PORT=$port, PATH=$path\n"; - + $timeout ||=180; + my $hard_timeout = time() + $timeout; require IO::Socket; local($^W) = 0; my $sock = IO::Socket::INET->new(PeerAddr => $host, @@ -174,8 +175,15 @@ my $buf = ""; my $n; - 1 while $n = sysread($sock, $buf, 8*1024, length($buf)); - return undef unless defined($n); + while( 1 ) { + my $remaining = $hard_timeout - time(); + return undef if $remaining <= 0; + if ( can_read( $sock, $remaining ) ) { + $n = sysread($sock, $buf, 8*1024, length($buf)); + return undef unless defined($n); + last if $n == 0; # we are eof + } + } if ($buf =~ m,^HTTP/\d+\.\d+\s+(\d+)[^\012]*\012,) { my $code = $1; @@ -191,6 +199,15 @@ } return $buf; +} + +sub can_read { + my($sock, $timeout) = @_; + my $fbits = ''; + vec($fbits, fileno($sock), 1) = 1; + my $nfound = select($fbits, undef, undef, $timeout); + die "select failed: $!" unless defined $nfound; + return $nfound > 0; } Administrator@JAMES /cygdrive/d



Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://368490]
[Your Mother]: Corion: I have lost the link to the pmdev guide(s). Do you have that handy?
[karlgoethebier]: Discipulus: see per lop please how to avoid the classic <c>perl 'rm -rfv *|'</>
[Eily]: karlgoethebier: more like grata non persona :P
Discipulus personae non gratae..
[Corion]: Your Mother: Hmm - I have the pmdev nodelet, which has some links, but I'm not sure what links/guides there are
[Corion]: pmdev wiki maybe?
[Discipulus]: uh neat karl!

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (8)
As of 2017-07-21 08:15 GMT
Find Nodes?
    Voting Booth?
    I came, I saw, I ...

    Results (319 votes). Check out past polls.