Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Simplest Forking

by Anonymous Monk
on Apr 08, 2014 at 16:00 UTC ( [id://1081522]=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I have 2 files, each split into 10 equal parts. I need to run an external C script with each part as an input (keeping parts of two files separate) and parse and combine the results for the entire file. Simplified example;

foreach my $Index ( 0 .. 10 ) { `Script Input:File_1[$Index], Paramater, Output: 'Results_File_1_'.$In +dex`; @Results_1 = Parse the 'Results_File_1_'.$Index using subroutine and a +ppend to @Results_1; `Script Input:File_2[$Index], Paramater, Output: 'Results_File_2_'.$In +dex`; @Results_2 = Parse the 'Results_File_2_'.$Index using subroutine and a +ppend to @Results_2; } Do something with @Results_1, @Results_2 (both have to finished).

I was wondering if there is an easiest way to fork the processes. I am not familiar with forking and I was thinking about running two processes at once in the background, waiting for them to finish and parse one after another (I am not sure if the appending will not clash if by chance two processes appending to same @Result will try to do it at once). Also I am not sure how to catch errors from the C script (any error should end the script printing the error message). Any suggestions? Many thanks!

Replies are listed 'Best First'.
Re: Simplest Forking
by Anonymous Monk on Apr 08, 2014 at 18:05 UTC
    Seriously consider whether multi-threading will hurt or will hinder a process like this. If you are processing a file, then this is an I/O-bound activity: the completion rate depends on how fast the machine can perform I/O, not on the speed of the CPU. Therefore, unless there is truly reasonable opportunity to overlap, the multi-threaded version might run noticeably slower.
      Thanks. When I try to do it using exec and then waitpid it does show substantial speed improvement. I however don't know how to deal with error zombie processes etc.
Re: Simplest Forking
by Anonymous Monk on Apr 08, 2014 at 22:36 UTC
    Proc::Background for simplest backgrounding of processes, wait for them to finish, kill them if they run too long ... whateveryouwant
Re: Simplest Forking
by Anonymous Monk on Apr 08, 2014 at 16:30 UTC

    Also if possible could you let me know how to detect how many threads are available (taking into account very unlikely single CPUs) and use of available ones?

    Also should I worry about zombie processes?

Re: Simplest Forking
by Anonymous Monk on Apr 09, 2014 at 01:05 UTC
    Doubt forking will do you a damm bit of good in this case ... so ... "f**k forking." Two processes running in parallel with each other can't process a single input-file as rapidly as a single process could do the same thing.
Re: Simplest Forking
by codiac (Beadle) on Apr 10, 2014 at 09:36 UTC
    Proc::Queue for forking & IPC::Run for calling the C script.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1081522]
Front-paged by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others browsing the Monastery: (7)
As of 2024-04-23 17:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found