Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Re: how do I create parrallel processing in this script?

by tirwhan (Abbot)
on Feb 05, 2010 at 09:40 UTC ( #821526=note: print w/replies, xml ) Need Help??

in reply to how do I create parrallel processing in this script?

Additionally to moritz's correct answer, I'll caution that you are unlikely to experience a huge difference in processing time by parallelizing the input operation. IO is limited by the disk speed, and turning a single process reading in one file at a time into several processes that read these files in parallel will likely affect performance detrimentally, because the disk head has to constantly jump to a new position while reading in. So, unless you have a very unusual drive configuration (or weird limits set on your single process by the OS), this will probably slow down the entire process, rather than speeding it up.

Also, reading in the file into an array in a forked child and then printing that array in the parent won't work, data structures don't remain shared in a forked process, you need some form of IPC for that.

All dogma is stupid.

Replies are listed 'Best First'.
Re^2: how do I create parrallel processing in this script?
by avanta (Beadle) on Feb 05, 2010 at 10:17 UTC
    Thanks a lot for the suggestion. I think I would be needing another help, in this code i wanted to read files and split into threads the process and then join the output of the threads into one file. How an I do it other than using fork(),

      Hmm. As I said above, there's not much point in doing this, because you almost certainly won't improve performance. However, if you did want to do this you could use one of the following methods:

      - print to the output file from the children instead of the parent. You'll need to lock the output file in some way (see "perldoc -q lock" for some ways on doing that) so that the writes are not completely jumbled.

      - use some form of Inter Process Communication to read the file content into the parent process. Read the documentation I linked to or search cpan for IPC for various ways of doing this.

      - use threads and a shared variable instead of fork.

      All dogma is stupid.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://821526]
[marto]: good morning all
[Corion]: Hi marto!
[Corion]: The fun show at $work continues, as The Big Project is now in its second week of frantic live-bugfixing and weekend releases where nobody knows what went live. Nothing has been tested anyway.
erix mutters cantankerously under his breath
Corion watches from the sidelines. Or rather, from behind, as my system only gets output from that process and my programs adhere strictly to the GIGO design principle.
[erix]: ah, that's nice to hear Corion :)
[Corion]: erix: Yeah, the sad thing is that all I can do is document things, so I can point fingers when the auditors come :-/
[Corion]: "I'm here to open tickets and point fingers. And I'm all out of tickets."
[erix]: didn't Sybase have pretty good auditing? :) (this is a vague memory)

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (8)
As of 2017-03-28 08:55 GMT
Find Nodes?
    Voting Booth?
    Should Pluto Get Its Planethood Back?

    Results (328 votes). Check out past polls.