Beefy Boxes and Bandwidth Generously Provided by pair Networks RobOMonk
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Multi Threading in perl sample code

by shan_emails (Beadle)
on Nov 05, 2012 at 05:40 UTC ( #1002275=perlquestion: print w/ replies, xml ) Need Help??
shan_emails has asked for the wisdom of the Perl Monks concerning the following question:

Hi All,

I am new to perl and I have the below task
1) read the config from oracle10g DataBase table (each config has a one process file)
2) get the files for processing
3) logs the processing comments in a log file

This config has the total of 2500.
So after reading from Database and get the files so totally we have 2500 files, now split it to 5 parts and then process this 5 parts at simultaneously. And keep logs of all 5 threads in a single log file.

kindly give some sample code for this threading in perl.

I am using HP-UX in Unix Operating System.

Thanks,
Shanmugam A.

Comment on Multi Threading in perl sample code
Re: Multi Threading in perl sample code
by Corion (Pope) on Nov 05, 2012 at 08:25 UTC

    My advice is to avoid using threads, especially as you are new to Perl.

    It is unlikely that processing the logs in threads will be much faster than processing the logs from a single threaded program launched as multiple instances. Most of the time of your programm will likely be spent on disk IO anyway.

    I recommend writing the program that processes a single log file first. Then I would launch that single program through runN, which will process all the files in parallel with it.

    As for logging, I recommend writing the output to a separate log file for every input file, and merging these log files after all files have been processed.

Re: Multi Threading in perl sample code
by Anonymous Monk on Nov 05, 2012 at 08:37 UTC
Re: Multi Threading in perl sample code
by Anonymous Monk on Nov 05, 2012 at 13:22 UTC
    Write one script if necessary that builds a to-do list. Then, write one script that does the job in sequence. Finally, from the command line or the shell, spawn n copies of that second script. A program is not an entire job.
Re: Multi Threading in perl sample code
by BrowserUk (Pope) on Nov 05, 2012 at 13:45 UTC
    now split it to 5 parts and then process this 5 parts at simultaneously.

    Why 5 parts? (Rather than 4 or 8 or ... )


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    RIP Neil Armstrong

Re: Multi Threading in perl sample code
by sundialsvc4 (Monsignor) on Nov 05, 2012 at 15:20 UTC

    Since you’re using Unix, do it “the Unix way.”   Write a script that does its part, then invoke as many parallel-executing copies of that script from another script or the command-line.   The program does not take on all the fancy-pants work of doing everything:   the shell does; the user does.   Each instance of the worker-bee (Perl) script simply finds another unit of work to do and does it, running until it finds no more work to do.   It has no awareness of any program other than itself.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1002275]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (6)
As of 2014-04-20 19:17 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (486 votes), past polls