That is indeed the correct solution (to use sysread). However, it's worse than simply because there's no newline.
select generally doesn't work properly if you mix in buffered reads and writes; select will perpetually think something is readable, or nothing is readable, because the buffering has over-read or under-read on the handle.
This means that whenever you're using select, either directly or indirectly through IO::Select or some event loop, then you must use sysread and syswrite (not print, <$fh>, read(), etc.).
You often end up doing your own buffering with sysread and syswrite, because often you're still dealing with line-oriented protocols. This is fairly boilerplate stuff, and it's recommended to use an event loop that does most of it for you (e.g. POE, Event, IO::Async, etc.).