Re: Will a shared filehandle processed by multiple threads/processes cause problems?

[wrog]

Two problems:

1. buffering,
   i.e., you may think you're only reading one line but underneath the hood you may be slurping up a lot more and there's no real way to control how much or to guarantee that what you're getting will end at a line boundary. The only way out of that box is to use the low-level operations (sysread, sysseek), at which point you're giving up the speed advantages of buffering and probably making many more system calls (sysseek) in order to leave the filepointer exactly at the end of a line
2. synchronization
   if two processes attempt to read or seek the same handle at the same time, all bets are off as to what will actually be read by either or where the seek pointer will end up. You will have to do some kind of locking to ensure only one process is reading at a time.

Much depends on how big the individual lines are vs. how much computation needs to be done on each one. If the latter is what's killing you and you have a true multicore processor (i.e., where multiple processes really can run simultaneously) then this approach could indeed win. If, on the other hand, you're mostly I/O-bound, i.e., reading+writing are what's taking up your time, then probably not.

You may want to consider having one process do all of the reading, giving it outgoing pipes to the other processes, which then all pull lines from the reader on demand (which then involves more games with signals or semaphores), which then gives you benefits of buffering and concurrency. This will be worth it if the lines are generally way smaller than the buffer size.

[alanraetz]

Thanks for the response, makes sense. I think any perl code that depends on assumptions about what happens under the hood is bad practice so I will avoid this. But wanted to throw it out there to get feedback, thanks.