Multi Threading in perl sample code

shan_emails has asked for the wisdom of the Perl Monks concerning the following question:

Hi All,

I am new to perl and I have the below task
1) read the config from oracle10g DataBase table (each config has a one process file)
2) get the files for processing
3) logs the processing comments in a log file

This config has the total of 2500.
So after reading from Database and get the files so totally we have 2500 files, now split it to 5 parts and then process this 5 parts at simultaneously. And keep logs of all 5 threads in a single log file.

kindly give some sample code for this threading in perl.

I am using HP-UX in Unix Operating System.

Thanks,
Shanmugam A.

Comment on Multi Threading in perl sample code

Replies are listed 'Best First'.
Re: Multi Threading in perl sample code by Corion (Patriarch) on Nov 05, 2012 at 08:25 UTC
My advice is to avoid using threads, especially as you are new to Perl. It is unlikely that processing the logs in threads will be much faster than processing the logs from a single threaded program launched as multiple instances. Most of the time of your programm will likely be spent on disk IO anyway. I recommend writing the program that processes a single log file first. Then I would launch that single program through runN, which will process all the files in parallel with it. As for logging, I recommend writing the output to a separate log file for every input file, and merging these log files after all files have been processed.	[reply]
Re: Multi Threading in perl sample code by Anonymous Monk on Nov 05, 2012 at 08:37 UTC
kindly give some sample code for this threading in perl. I could have sworn there was an example perlthrtut ... maybe these will help Perl Threads Boss/Worker Example thread/fork boss/worker framework recommendation threads: work crew memory leak WorkCrew threading Re: TCP Server hangs with multiple client connections, Thread::Queue	[reply]
Re: Multi Threading in perl sample code by BrowserUk (Patriarch) on Nov 05, 2012 at 13:45 UTC
now split it to 5 parts and then process this 5 parts at simultaneously. Why 5 parts? (Rather than 4 or 8 or ... ) With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP Neil Armstrong	[reply]
Re: Multi Threading in perl sample code by sundialsvc4 (Abbot) on Nov 05, 2012 at 15:20 UTC
Since you’re using Unix, do it “the Unix way.” Write a script that does its part, then invoke as many parallel-executing copies of that script from another script or the command-line. The program does not take on all the fancy-pants work of doing everything: the shell does; the user does. Each instance of the worker-bee (Perl) script simply finds another unit of work to do and does it, running until it finds no more work to do. It has no awareness of any program other than itself.
Re: Multi Threading in perl sample code by Anonymous Monk on Nov 05, 2012 at 13:22 UTC
Write one script if necessary that builds a to-do list. Then, write one script that does the job in sequence. Finally, from the command line or the shell, spawn n copies of that second script. A program is not an entire job.	[reply]

Back to Seekers of Perl Wisdom