Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask

Perl and parallel processors

by Anonymous Monk
on Apr 24, 2007 at 12:38 UTC ( #611713=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hello all!
I know nothing about Perl and pcs with parallel processors and I wanted to ask if Perl supports this feature. We have a system of 20parallel processors in my University and I must write my perl script in a way that will take advantage of as many proccesors as it can...
Do I need to write something inside the code?
Any hints would be greatly appreciated...

Replies are listed 'Best First'.
Re: Perl and parallel processors
by Joost (Canon) on Apr 24, 2007 at 13:06 UTC
    Perl does nothing with parallel processors by itself (AFAIK). If your algorithm lets you, probably the easiest way to get some extra efficiency on multi-processor systems is to fork() off a bunch of processes that work together. Parallel::ForkManager might be useful.

    Another option is to use threads but working with threads in perl isn't as straight-forward as you might imagine. Especially don't expect to be able to share objects from classes that weren't written to be thread-aware (which probably includes most modules on CPAN).

    For numeric processing PDL supports something called PDL threads, but exactly how that relates to multi-processors is foggy to me.

      Hi Joost - just a quick note to point out that PDL threading isn't (yet, as far as I know) something that lets you use multiple processors.

      It's probably easiest to just think of a PDL thread as simply an optimised loop: instead of using a loop in Perl, it lets you use a loop in PDL's C code, which is of course quite a lot quicker.

      The docs say:

      why threading ?

      Well, code that uses threading should be (considerably) faster than code that uses explicit for-loops (or similar perl constructs) to achieve the same functionality. Especially on supercomputers (with vector computing facilities/parallel processing) PDL threading will be implemented in a way that takes advantage of the additional facilities of these machines. Furthermore, it is a conceptually simply construct (though technical details might get involved at times) and can greatly reduce the syntactical complexity of PDL code (but keep the admonition for documentation in mind). Once you are comfortable with the threading way of thinking (and coding) it shouldn't be too difficult to understand code that somebody else has written than (provided he gave you an idea what exspected input dimensions are, etc.). As a general tip to increase the performance of your code: if you have to introduce a loop into your code try to reformulate the problem so that you can use threading to perform the loop (as with anything there are exceptions to this rule of thumb; but the authors of this document tend to think that these are rare cases ;). (emphasis mine)

      Best wishes, andye

Re: Perl and parallel processors
by jettero (Monsignor) on Apr 24, 2007 at 13:04 UTC

    I went looking for things like this a while back. I didn't turn up all that much that I was tempted to use or design a project around. Although, there are some packages you can try if you're already really excited about parallel programming in perl.

  • The State of Parallel Computing in perl 2007?

    There were some really helpful answers in that thread.


Re: Perl and parallel processors
by Moron (Curate) on Apr 24, 2007 at 13:24 UTC
    There are two types of multiprocessing architecture: one where the programmer has to specifically schedule jobs on different machines and one where a configurable operating system chooses the processor based usually on a load-balancing algorithm.

    We need to know which type of architecture and even more detail such as the operating system and how it achieves multiprocessing (virtual in a network or is it a Cray or an equivalent HP or Sun Unix/Linux purpose-built multiprocessing computer) before we can advise fully, although Parallel::ForkManager is an example of an implementation where the operating system chooses the processor without needing scheduling from the programmer.

    Further, it isn't clear whether you already have a specific exercise to carry out or whether you also need us to dream up a functional example for you.

    The most useful and simple example I can think of comes from my own commercial experience: you have a group of a few hundred accounts each with tens of thousands of entries for today. Given that you have a farm of 64 linux PCs connected together as a virtual multiprocessing machine, calculate today's profit or loss for the group of accounts as quickly as possible (a.k.a "end of day" accounting - in real investment banking it isn't so simple because of complex pricing rules but can be made dirt simple for an exercise). Update, i.e. in the simplified case, an entry has an account number say 001-500, a unique transaction id, a debit/credit flag and an amount greater >= 0.0. Share the accounts among the processors to produce P/L per account and a grand total P/L value.


    ^M Free your mind!

    A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Perl and parallel processors
by moklevat (Priest) on Apr 24, 2007 at 15:27 UTC
    You can certainly do parallel processing using perl. The specific implementation depends a lot on the problem you are trying to solve.

    Whatever the situation, your University's cluster/grid probably has some tools installed to make parallel processing simpler. Ask around or check for PVM, LAM/MPI, or MPICH2. Learn how these systems work and check out CPAN for the perl modules that interface with them. Parallel::PVM and Parallel::MPI are both available, but the PVM module has seen more recent updates and has a higher version number, so it might be a better bet.

    Good luck with the project.

      There now exist a new module called Many-core Engine (MCE) for Perl. There are many examples including matrix multiplication with PDL across many cores.

Re: Perl and parallel processors
by sgifford (Prior) on Apr 24, 2007 at 15:53 UTC
    If your architecture supports it, you could also try the Parallel::MPI interface to MPI, a message-passing interface. I have used MPI from C, and it's quite fast and efficient.
Re: Perl and parallel processors
by explorer (Chaplain) on Apr 24, 2007 at 13:07 UTC
Re: Perl and parallel processors
by diotalevi (Canon) on Apr 24, 2007 at 22:32 UTC

    I've recently been integrating Oz with Perl. Among other things, Oz is a multi-threaded interpreter. It'd be possible to export control or compute time to Oz and gain parallelism .

    ⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://611713]
Approved by Corion
Front-paged by Corion
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (3)
As of 2022-05-17 20:28 GMT
Find Nodes?
    Voting Booth?
    Do you prefer to work remotely?

    Results (68 votes). Check out past polls.