The basic problem is that the terminal device driver doesn't consider a read to have happened until the user presses a return character - this means that no program is going to be able to read a single character unless they first tell the device driver: "Behave differently, and consider a read complete after a single character".
Hence the suggestion to use Term::ReadKey - it tells the device to behave differently. At a C level, you'd use tcsetattr out of the termios function set. If for some reason you don't think that your deployment environment will have Term::ReadKey installed, you could try a system call to stty; "stty cbreak" will enter the one-character-at-a-time mode and "stty icanon" will take you back. (see stty's man page)
By the way, on Win32 the way I'd approach this is though the InputChar function inside Win32::Console.