Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Perl and parallel processors

by Anonymous Monk
on Apr 24, 2007 at 12:38 UTC ( #611713=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hello all!
I know nothing about Perl and pcs with parallel processors and I wanted to ask if Perl supports this feature. We have a system of 20parallel processors in my University and I must write my perl script in a way that will take advantage of as many proccesors as it can...
Do I need to write something inside the code?
Any hints would be greatly appreciated...

Comment on Perl and parallel processors
Re: Perl and parallel processors
by jettero (Monsignor) on Apr 24, 2007 at 13:04 UTC

    I went looking for things like this a while back. I didn't turn up all that much that I was tempted to use or design a project around. Although, there are some packages you can try if you're already really excited about parallel programming in perl.

  • The State of Parallel Computing in perl 2007?

    There were some really helpful answers in that thread.

    -Paul

Re: Perl and parallel processors
by Joost (Canon) on Apr 24, 2007 at 13:06 UTC
    Perl does nothing with parallel processors by itself (AFAIK). If your algorithm lets you, probably the easiest way to get some extra efficiency on multi-processor systems is to fork() off a bunch of processes that work together. Parallel::ForkManager might be useful.

    Another option is to use threads but working with threads in perl isn't as straight-forward as you might imagine. Especially don't expect to be able to share objects from classes that weren't written to be thread-aware (which probably includes most modules on CPAN).

    For numeric processing PDL supports something called PDL threads, but exactly how that relates to multi-processors is foggy to me.

      Hi Joost - just a quick note to point out that PDL threading isn't (yet, as far as I know) something that lets you use multiple processors.

      It's probably easiest to just think of a PDL thread as simply an optimised loop: instead of using a loop in Perl, it lets you use a loop in PDL's C code, which is of course quite a lot quicker.

      The docs say:

      why threading ?

      Well, code that uses threading should be (considerably) faster than code that uses explicit for-loops (or similar perl constructs) to achieve the same functionality. Especially on supercomputers (with vector computing facilities/parallel processing) PDL threading will be implemented in a way that takes advantage of the additional facilities of these machines. Furthermore, it is a conceptually simply construct (though technical details might get involved at times) and can greatly reduce the syntactical complexity of PDL code (but keep the admonition for documentation in mind). Once you are comfortable with the threading way of thinking (and coding) it shouldn't be too difficult to understand code that somebody else has written than (provided he gave you an idea what exspected input dimensions are, etc.). As a general tip to increase the performance of your code: if you have to introduce a loop into your code try to reformulate the problem so that you can use threading to perform the loop (as with anything there are exceptions to this rule of thumb; but the authors of this document tend to think that these are rare cases ;).
      http://pdl.sourceforge.net/PDLdocs/Indexing.html#threading (emphasis mine)

      Best wishes, andye

Re: Perl and parallel processors
by explorer (Chaplain) on Apr 24, 2007 at 13:07 UTC
Re: Perl and parallel processors
by Moron (Curate) on Apr 24, 2007 at 13:24 UTC
    There are two types of multiprocessing architecture: one where the programmer has to specifically schedule jobs on different machines and one where a configurable operating system chooses the processor based usually on a load-balancing algorithm.

    We need to know which type of architecture and even more detail such as the operating system and how it achieves multiprocessing (virtual in a network or is it a Cray or an equivalent HP or Sun Unix/Linux purpose-built multiprocessing computer) before we can advise fully, although Parallel::ForkManager is an example of an implementation where the operating system chooses the processor without needing scheduling from the programmer.

    Further, it isn't clear whether you already have a specific exercise to carry out or whether you also need us to dream up a functional example for you.

    The most useful and simple example I can think of comes from my own commercial experience: you have a group of a few hundred accounts each with tens of thousands of entries for today. Given that you have a farm of 64 linux PCs connected together as a virtual multiprocessing machine, calculate today's profit or loss for the group of accounts as quickly as possible (a.k.a "end of day" accounting - in real investment banking it isn't so simple because of complex pricing rules but can be made dirt simple for an exercise). Update, i.e. in the simplified case, an entry has an account number say 001-500, a unique transaction id, a debit/credit flag and an amount greater >= 0.0. Share the accounts among the processors to produce P/L per account and a grand total P/L value.

    __________________________________________________________________________________

    ^M Free your mind!

      I think that in many cases an agent based system works really well. The agent can then choose whether the machine that it is sitting on is ready to take the request. The agents compete for the requests coming in or the user can decide whether to direct a request to a specific machine (and processor). I've implemented (with the help of Perl monks) a system like this and all sorts of additional features can be added.
        1) I expect the choice of multiprocessing architecture to be OT rel. OP, not least because it will be a given as to what it is.

        2) I suggest you find out what a cluster is rather than try to reinvent what it might have been in a parallel universe. Take a look also at the theory and practice of parallel computing

        __________________________________________________________________________________

        ^M Free your mind!

Re: Perl and parallel processors
by moklevat (Priest) on Apr 24, 2007 at 15:27 UTC
    You can certainly do parallel processing using perl. The specific implementation depends a lot on the problem you are trying to solve.

    Whatever the situation, your University's cluster/grid probably has some tools installed to make parallel processing simpler. Ask around or check for PVM, LAM/MPI, or MPICH2. Learn how these systems work and check out CPAN for the perl modules that interface with them. Parallel::PVM and Parallel::MPI are both available, but the PVM module has seen more recent updates and has a higher version number, so it might be a better bet.

    Good luck with the project.

      There now exist a new module called Many-core Engine (MCE) for Perl. There are many examples including matrix multiplication with PDL across many cores.

      https://metacpan.org/module/MCE

Re: Perl and parallel processors
by sgifford (Prior) on Apr 24, 2007 at 15:53 UTC
    If your architecture supports it, you could also try the Parallel::MPI interface to MPI, a message-passing interface. I have used MPI from C, and it's quite fast and efficient.
Re: Perl and parallel processors
by diotalevi (Canon) on Apr 24, 2007 at 22:32 UTC

    I've recently been integrating Oz with Perl. Among other things, Oz is a multi-threaded interpreter. It'd be possible to export control or compute time to Oz and gain parallelism .

    ⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://611713]
Approved by Corion
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (7)
As of 2014-08-23 14:14 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (174 votes), past polls