If you really need to shave off some speed, you might want to rethink your approach. I know neither your underlying data source nor your protocol, but you should check whether you really can only get one byte at a time, since you seem to be processing the data in blocks anyway.
A common approach is to read a block of n bytes from the serial port/keyboard/.... If the routine you use to read is blocking (ie., it won't return until it has collected n bytes), you can use a timeout to make sure you don't wait forever.