http://www.perlmonks.org?node_id=1090656

doubleqq has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I am trying to run a script on a SGE (qsub, etc) computing cluster. I am getting a little confused about the various Perl modules available and hope someone can help me sort it out. It seems the module Parallel::Forkmanager allows for the management of multiple threads on a given core, which is handy but not quite what I am looking for.

Is there a Perl module that allows me to run a series of processes in parallel across several nodes on our cluster? I took a look at the other Parallel modules, and none seemed quite right, I am not using a virtual machine or having issue with my subroutines. Basically I have 30 nodes, each node can handle 20 threads. As I understand it, with Parallel::Forkmanager I can run my script using the 20 threads on a 1 specific node, but what I really want to do is run my script across maybe 5 nodes using 10 threads each (for example, ideally I would like the script to make best determination). Am I just not understanding Parallel::Formanager, or were there any other modules specifically for this purpose?

Thank you for any insight!

Replies are listed 'Best First'.
Re: HPC Computing Question
by atcroft (Abbot) on Jun 20, 2014 at 18:37 UTC

    A quick search of CPAN for HPC returned GRID::Cluster and GRID::Machine that look more like what I believe you were expecting. (A search for cluster was apparently a little too wide a net for a quick search, however (although there may be something useful there as well).)

    Hope that helps.

Re: HPC Computing Question
by Laurent_R (Canon) on Jun 20, 2014 at 18:45 UTC
    You might want to take a look at PVM (Parallel Virtual Machine) and MPI, two message passing systems for parallel processing. Both have Perl modules to use them. I have used both about 15 years ago and they worked fairly well (but I haven't used the related Perl modules as I was using them with C and Fortran programs at the time).
Re: HPC Computing Question
by perlfan (Vicar) on Jun 21, 2014 at 17:17 UTC
    I was a scientific computing consultant for 4 years at LSU and LONI a while back. Systems included Linux clusters (PBS/Torque) and IBM Power systems (LoadLeveler). We not only participated in research and helped users port and optimize their codes on various systems, but we also maintained (and evaluated potential) systems.

    First, I would say that there is no Perl module that allows one to manage jobs across nodes in the cluster as a set of cooperating sequential processes; not the way you're thinking anyway.

    Let's take a step back and remove Perl from this equation. There are basically 2 different approaches to parallelizing your codes. First, there is pure message passing though libraries that meet the Message Passing Interface (MPI) specification. This library provides for a standard programming interface and execution environment for some number of communicating sequential processes (both logical processes and literal Unix processes). MPI provides data type primitives, means of communications, and synchronization constructs to orchestrate the computation you're coordinating among the CSPs. This is inter-node communication.

    You also have OpenMP (for example) for a well standardized shared memory programming (SMP) interface. This is used to write tightly coupled, threaded code on a single "CCNUMA" node. This is intra-node communication. Most C/C++ and Fortran compilers supported OpenMP. For more information on it, I have an OpenMP tutorial I created a few years ago.

    Still keeping Perl out of this, what you want to do sounds like the third option - i.e., the "hybrid" approach. What you need to do here is basically have an SMP (threaded single process) program running on each node using something like OpenMP to thread the program; each of these processes communicate across the nodes using some sort of message passing, like MPI. Hybrid MPI/OpenMP programs are a fairly common target for playing and testing, but I know of only a few that actually take advantage of this rich and interesting approach.

    To bring Perl back into it here, there is nothing that is going to give you this hybrid approach for free. You will need a Perl based way of passing messages among the nodes. On each node, you will need a way to manages the "threads" that are running. I would strongly recommend that in an HPC environment you look at MPI & OpenMP, even if it is to get familiar conceptually with what it takes to create a hybrid application. You should also look at the MPI interfaces and threading interfaces PDL provides. There has been a lot of work to offer access to parallelization technologies through PDL; but frankly nothing is going to be as good (or interesting) as use C/C++ or Fortran + MPI&OpenMP.

    I contributed to a lot of the documentation at LSU and LONI regarding OpenMP and writing hybrid codes, for example here one would run a hybrid job on one of the Linux clusters. There is also information on that site, along with many other sites (some better than others) on writing your first hybrid code.

    Like I said, it's a rich and interesting world. I highly encourage you to look at this and understand the paradigms. Once you do, it'd be fairly straightforward to translate the concepts using any number of intra-node communication and inter-node threading technologies.

    I used a lot of Perl for managing the systems and running codes - some mine, some others. I also translated a computational molecular dynamics code one friend of mine had written into a Perl version, just for fun. If you have any questions regarding Perl and HPC just post them here and I'll reply if I can help. If you have strictly HPC related questions, you can PM me.

    Other Perl and HPC resources include the PDL list/resources and the Quantified Onion list. There is an HPC BoF at YAPC this week that I plan on attending, so I know that while the HPC community in Perl is small, it is very much alive.

      To the best of my knowledge, there isn't yet any PDL MPI interface. The closest to working is PDL::SVDSLEPc, but that has (per moocow) never been used in production. However, there are plans afoot (see https://github.com/PDLPorters/pdl/issues/349) to use Alien::OpenMP to get OpenMP support going in PDL.

      Once that is in place, it will make sense to start thinking about use of MPI. A hint of what's required is that GPUs require passing data to and from them, and marshalling execution, which resembles MPI somewhat.

Re: HPC Computing Question
by monkey_boy (Priest) on Jun 21, 2014 at 11:31 UTC
    I think you are barking up the wrong tree. I haven't used SGE for a number of years, but in its earlier incarnations, there was no concept of shared threads/memory BETWEEN nodes. The usual way you utilize the nodes is to write code such that smaller sections of data can be calculated independently. A usuall example is a script that takes a data file, then does some calculations upon it. Then, if we have 100 data files (e.g. numbered 1.txt .. 100.txt), we can:
    qsub -t 1-100 myscript.pl
    An in the code
    open (my $fh , "<", $ENV{SGE_TASK_ID}.".txt") or die $!; etc ...
    SGE then in your case would divide each task to a node as it sees fit (dependent on how the SGE is configured). Hope this helps.


    This is not a Signature...