Socket dilemma - sending EOF

[Anonymous Monk]

I've been successfully using IO::Socket for quite some time now to communicate between Java & Perl servers. I always use shutdown to send an EOF to the remote server so that it knows I've finished writing:

$sock->shutdown(1);

Just recently I needed to connect to another third party via a socket connection. For some reason whenever I close the write portion of the socket (via $sock->shutdown(1)) the third party server closes the socket and therefore I cannot read their response. If there any other way to let the remote server know I've finished writing data i.e sending EOF *without* using shutdown(1)??

The only suggestion the third party had was "why don't you just flush the buffer?".

Any help you could provide would be super.

Thanks.

John

[Nuke]

  My understanding is that shutdown(1) sends the "FIN" segment, which signals the other end that you are finished writing data. The other end should send back an "ACK" segment and perform a passive close on the socket.

  What happens next is the other end sends back a "FIN" segment of it's own. Your end then sends back a final "ACK", and the socket is completely closed.

  The confusing part is why you are using shutdown if you are expecting more data on the line. The model of socket communication I have seen designates that you don't send a "FIN" until you've received the data you want. (This allows more than one request per connection.)

  It sounds to me like the other end is enterpreting the initial "FIN" as the end of all communication, and is dropping any data that hasn't been sent yet. Whether this is happening on the socket level, or the application level, I couldn't tell you without seeing the code on the other end. To get around it, just hold off on using shutdown until you've received the data you requested, or until the other end initiates a shutdown.

  In a perfect world, every socket connection would follow the specification to the letter, but it doesn't happen. The above method should still work for both of your existing socket connections as well if you need to use just one bit of code for all three connections.

  I hope that helps...



  Update: I just checked W. Richard Stevens tome of sockets knowledge; "Unix Network Programming", and my thoughts are confirmed. In order to follow sockets specifications you shouldn't shutdown until you've received your requested data, or the other side begins the socket shutdown process. The fact that your original code works with the other two socket connections just means that they have coded their application to flush all data before performing the final close of the socket. A good idea, but not necesarily required.


  Update2: I guess I'm saying there should be some query string you send across the socket that warrents a response from the other end. Closing the socket is a poor way to signal you're done sending/requesting data, but it's used often enough that I guess we have to live with it.


Nuke
[e-mail]
www.nuke3d.com

[sfink]

Stevens doesn't say shutdown-based transactions violate good practices, I don't think.

shutdown(1) forces half of the connection to be closed. I often use this for the same purpose as the original poster. The other end has no obligation to close the other half when this happens. Stevens is talking about close(), which is a different beast. It decrements a reference count, and if it reaches zero, shuts down both halves of the connection. shutdown doesn't bother with reference counts; it immediately forces one or both halves shut.

You should never close a connection before reading out all of the data (that'll lock up the port for a very long time and may send a broken pipe signal to the other end.) I assume the same thing would happen if you did shutdown(0) with data in transit. But it's fine to shutdown(1) the half you're writing to; it's then the problem of the other end to read everything out and take that half down.

But in answer to the original question, you're not going to be able to send an intermediate EOF. You'll either have to change the protocol (4-byte network byte-order chunk sizes work fine, with zero meaning end-of-message), or make new connections. There are nasty little wrinkles in TCP that you could try to exploit (eg out of band data), but that'll do you much more harm than good. And it's still a protocol change, just at a different level.

But I don't really understand what was meant by "the third party server closes the socket and therefore I cannot read their response". As long as they write the full response to the socket before calling close(), it'll be fine -- you can read their response, get an EOF, and do a close() yourself. Are you getting a broken pipe? What happens?

Just for illustration, here's a simple shutdown-based handshake (yes, I just wanted to see if I could do it).perl -MIO::Socket -we 'sub S{$l=IO::Socket::INET->new(LocalPort=>9123,Listen=>5);$s=$l->accept();print "rS:$_" while(<$s>); print $s "what?\n";} sub C{sleep 1;$c=IO::Socket::INET->new(PeerAddr=>"localhost:9123"); print $c "hello\n"; shutdown($c,1); print "rC:$_" while(<$c>);} exit(fork?S:C)'

Update: Wow, I was way too argumentative in my initial response. It looked like I was disagreeing with lots of stuff I fully agree with (eg how half-closes are realized in TCP packets). I rewrote the first sentence to be more specific.

[Nuke]

Good points. I'm not sure I totally agree. Shutdown(1) forces a close of half the socket. That's fine and inline with what I said. In order for this to happen, on the TCP level, a FIN must be sent to the other end. That end will answer back with an ACK of it's own, which completes the shutdown of HALF of the socket.


My next statement is a bit misleading I guess. You are correct that the other side does not have to initiate a shutdown of the other half of the socket right away. However, it's probably better to keep both halves of the socket open until you're done sending and receiving. This would allow multiple requests in a single session. That may not be required NOW, but experience has taught me that those who dictate what you need to code will oftimes ask for things you thought you'd never need.

But I guess it's moot, because the bottom line is the protocol is going to have to change, since the other end is apparently enterpreting the shutdown of half the socket as a hard close of the socket.

I'll admit to one thing... I've never actually used IO::socket, and am making some assumptions based on documentation I can find. If I'm way off, please tell me exactly how. I think I've got it right though.



Nuke
[e-mail]
<a href="http://www.nuke3d.com"=p:a.nuke3d.com

[rbc]

Have you tried doing the following?


#Before writting on the socket
$sock->autoflush(); # so output gets there right away




#when writting $data to the socket
print $sock $data . "\015\012";




#when done writting to the socket
close($sock);


Hope that helps :)

[wileykt]

I agree with rbc that you want to close the socket connection, but before you do that, try using something to flag the end of either the file or transmission. I've used both \02, \03 and \04 which are the hex equivalents of start of text, end of text and end of transmission. To look at these values just type man ascii in unix. Both the client and the server would have to use these as the transmission flags.