Try the unbuffered read() function instead of sysread(). Do not mix buffered and unbuffered reads! That will cause big trouble!
You should be able with wireshark to tell if the problem is in the receive buffering or in the transmit delay from the other end. I am suspecting however that there is some delay while processing these very short packets. Although the time frame seems to be incredibly long! Can you provide more timing info from wireshark? What happened when you did: $socket->flush(); after the write (in Perl lingo the "print"?
Update: I was confused.. corrected and also another suggestion posted.