sysread and syswrite in tcp sockets

[rustybar]

Hi I have used syswrite to a socket to send a file to the client size. The file size is around 10.5k. On the client size, I use sysread to read from socket to receive the file. But it always receive the files in 2 chunks. The first chunk is always 4344 of size, and the 2nd chunk is the rest of the size. I want to receive the file as 1 whole thing and not divided into 2 chunks. Please advise.

Thanks!

Client:

while($len = sysread($sock, $buffer, 500000000))
{

createFile("<filename>", $buffer, $len, 0);
        
#processFile;
}        


Server:

open READFILE, "<", "<filename>" or die $!;
                        binmode(READFILE);

while($len = sysread(READFILE, $buffer, 500000000)){
                           
syswrite($new_sock, $buffer, $len);
                            
}
                        close(READFILE) or die $!;

[gone2015]

If you use sysread you must expect partial reads -- see the fine documentation. (Update: sorry, the documentation doesn't tell you this... I felt sure it did... I feel sure it should. Update 2: I am grateful to brother ikegami for pointing out ( see below) the significance of the word Attempts, which I am ashamed to admit I had missed.)

You could try plain read, which is less "clever".

Or you wrap a loop around the sysread, eg:

  my $rc ;
  my $buff = '' ;
  do { $rc = sysread(FH, $buff, 5000000, length($buff)) ; } while ($rc
+) ;

which will suck away at FH until it hits eof ($rc == 0) or hits some error (!defined($rc)). (Noting that if the FH is set to be non-blocking you can get "soft" EAGAIN error(s)...)

[ikegami]

It is documented in the very first word: (emphasis mine)

Attempts to read LENGTH bytes of data into variable SCALAR from the specified FILEHANDLE

sysread returns what's available

• when the end of file is reached
• when there's fewer than LENGTH bytes available from a pipe or socket
• when the call is interrupted by a signal

None of those conditions are considered errors.

[rustybar]

Hi, Thanks for the reply. Actually for the client program, I did use print to send username and password to the server first for authentication before receiving the file. Will it help if I change print to syswrite instead? I did try before but it doesn't work too.

Updated Client program:

create_socket #function to establish connection with server

print $sock, <username>;
sleep(1);  #to allow a pause so that username and pwd are 2 streams
print $sock <pwd>;

while($len = sysread($sock, $buffer, 500000000))
{

createFile("<filename>", $buffer, $len, 0);
        
#processFile;
}

[gone2015]

I'm not sure I understand the problem here... You're printing the <username> to a socket, sleeping for 1 second, and then printing the <pwd>. I cannot tell whether the other end will see these as separate or not -- though it is likely if client and server are the same machine.

sysread will return what has been received so far, or will wait until at least 1 byte is available -- so if the receiver is fast enough it will effectively see each packet as it arrives (all the bytes of a packet become available at the same time).

read, on the other hand, will collect data from incoming packets until it has the number of bytes you asked for, or the connection is closed.

At the sender end we have: (a) any buffering done at the Perl level (in particular PerlIO); and (b) any buffering done at the socket level.

Stuff output by print will be buffered by PerlIO. If 'autoflush' is set, then at the end of a print statement, the buffer will be flushed to the socket. But syswrite bypasses the PerlIO buffering, and sends stuff directly to the socket. You will find that IO::Socket::INET sets 'autoflush' by default -- so by default the difference between print and syswrite is small, except that with syswrite you have to cope with partial writes !

Now, there's nothing in TCP that absolutely requires packets to be sent as soon as there is data ready to go, but generally it will do so unless data previously sent has not yet been acknowledged (this is Nagle's algorithm). So, whether the result of separate print or syswrite operations arrive together depends to some extent on the speed of the network.

As far as you fragment of code is concerned,

while($len = sysread($sock, $buffer, 500000000)) {
  createFile("<filename>", $buffer, $len, 0);
  #processFile;
}

this will (if it is fast enough) createFile() for each packet received... I suspect that isn't what you had in mind ?

[lakshmananindia]

Check your send and recv buffer size. That might be a problem. Did you used any "print","write", "seek", "tell", or "eof" in other parts of the program? Because the perlio or stdio layers usually buffers data.

[apl]

Rather than doing a sysread, I prefer a call to a local routine to do all of the sysreads necessary to receive a valid message (ala what oshalla suggested).

There's usually some sort of protocol involved in sending a file (a Start Of Buffer indicator and a length, at the very least before the start of data, and an End Of Buffer indicator at the end). If you receive a partial buffer, and then a second buffer that also starts with a valid SOB indicator, my inclination would be to throw the initial buffer away and log a warning.

Putting all the logic to read a message into a routine encapsulates it from the logic of the program that uses the message once it's received.

[rustybar]

would there be a simpler solution, such as clear the perlio or stdio layers as it might be buffered by the print statement before proceeding to sysread? Thanks!

[rustybar]

Hi guys,

thanks for all the inputs. my client and server are actually running on different machine.

from the input given by you all, i figured that my problem is because sysread uses partial read. thus when i send a big file size, it will partial read it. thus on my client side i have to buffer it myself so as to append the partial data together to form back the single file.

thus i make sure of select with timeout to help me in this. below is my working code, hope it will helps anyone who encounter this problem. once again, thanks for the help!

#sysread uses partial read, so a stream of file might be break down in
+to 2-3 chunks
#thus use select to establish a timeout, if there is an interval of 1 
+sec or more where no data
#coming in, then it is sate to say there will be a new data stream, as
+ my data is coming in interval of fixed n secs.

while(1){        
  $len = 0;
  do{ 
    $offset = 0;
    if(defined $buffer){
    $offset = length($buffer);
  }
  $temp = sysread($sock, $buffer, 500000000, $offset);
  $len += $temp;    #append data together if they come in back to back
} while ($select -> can_read(1));
#if > 1 sec without any data coming in, quit
        
createFile(<filename>, $buffer, $len, 0);

#processFile();

[dbooth]

It would be nice if the perl docs on sysread were clearer on the question of partial reads. The word "Attempts" is inadequate: of *course* sysread will *attempt* to read. After all, if the file contains fewer bytes than requested, or if the file is a terminal or some such, then an *attempt* to read more bytes than are available will result in a partial read.

But the perl docs should be clearer about whether sysread will ever read fewer bytes than requested in cases where those bytes *are* available to be read. As is, a cautious programmer must assume that partial reads could happen for no apparent reason, and from the above discussion this sounds like it is the case. But it would be nice if the documentation made this clear, both to help those who might not notice this subtle case and to avoid wasting the time of those who do notice and spend time trying to find the answer in the docs.

Anybody know where to make this suggestion? I didn't see any indication on the perlfunc page.

[ikegami]

But the perl docs should be clearer about whether sysread will ever read fewer bytes than requested in cases where those bytes *are* available to be read.

First of all, I don't see how the answer could be anything other they yes. There are countless layers involved. Presumably, one of them could always provide more bytes as long as the handle is still open.

In short, why does it matter when more data will come? You've got data to process.

[salva]

Anybody know where to make this suggestion?

On the Perl 5 Porters mailing list.