Re: Will a shared filehandle processed by multiple threads/processes cause problems?

[Athanasius]

Maybe I’m misunderstanding, but I don’t see why you need to share the filehandle among any of the child processes. Have the parent thread read in the file, split the data into appropriate-sized chunks (or lines, as you say), and feed each chunk to a different child process. This will avoid all the problems arising from shared filehandles, including those detailed by wrog, above, and without losing any of the potential benefits of utilising multiple cores. (Whether those benefits outweigh the additional overhead of creating and managing the child processes is another question, one which you will need to answer by profiling.)

Hope that helps,

[alanraetz]

Yes, this was my original thought, to split up the file first and then spawn a bunch of processes for each chunk... thanks for the response. Actually my thought now is to point the script to a test input file that takes about 10 seconds to process and either do some speed profiling or just direct code optimization and see if I can speed it up. There are numerous inefficiencies in the script, so I have some ideas.