Perl Monk, Perl Meditation | |
PerlMonks |
Re: Run a script in parallel modeby marioroy (Prior) |
on May 26, 2015 at 14:52 UTC ( [id://1127842]=note: print w/replies, xml ) | Need Help?? |
Update: Workers remove the input file after running. Update: Changed from FS to RS option. Update: The OP mentioned having a big file. Also, a sequence file. I added the FS option to chunk the input file by records, not by lines. This works quite well. A chunk size value of 100 means 100 records, not 100 lines. There are may possibilities with various modules on CPAN. Below, I describe a way using MCE. I can follow up with another post with a version which unlinks tmp files orderly while running if processing in the thousands. MCE::Signal provides a $tmp_dir location. MCE itself is a chunking engine. Thus, each chunk comes with a chunk_id value. The sprintf is used mainly to have ordered output from running cat *.out.
In Section
Seekers of Perl Wisdom
|
|