I know nothing about rotatelogs but I would assume that if it is used with the CustomLog directive that Apache executes it once at startup and then pipes all log output through this one process. If you simply substituted a pipe to it in your open call then you would be executing it every time mylog() gets called, this will lead to increased overhead, not to mention the possibility that rotatelogs won't work like you want if it is repeatedly run like this (it might, for example, start a new log each time it is run rather than appending to the previous one). These problems would be solved by opening it once at startup and then just printing to it in mylog().
in reply to open FH, "|foo" under modperl2
Yes, you can open the pipe like a file
flock only works on files (as opposed to filehandles), but if multiple processes are using the same pipe then you will probably need some sort of synchronization such as locking a sentinal file. When I say "using the same pipe" I am referring to creating the pipe outside of mylog() as mentioned earlier, but even if you were creating it in mylog() you might have multiple copies of rotatelogs writing to the same file depending on how rotatelogs works so a lock on a sentinal file would be a good idea here too.
Your last two questions were answered with the first.