Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re: distributed computing

by Roger (Parson)
on Aug 31, 2005 at 17:46 UTC ( #488163=note: print w/ replies, xml ) Need Help??


in reply to distributed computing

Welcome to the world of super computing! Your computers are linked in a NOW (network of workstations) I presume. Cluster computing with Perl is very easy. Let me explain it...

The ideal configuration for a beowulf cluster is to have a head node (a node is a computer) that has access to the external network, and a private network to link up the rest of the nodes with the head node. The head node will have two network cards, one for the internal cluster, one for communicating with the rest of the world.

C--+--C | C--+--C | C--Hub--C(head) | | ---------+----------- Backbone LAN
This is the ideal world, I doublt that you want to configure your cluster in this mannar.

The easier way is to connect all the computers via the back bone LAN into a NOW (network of workstations), no special hardware is required.
C C C C(head) | | | | | | | | -+--+--+--+---- Backbone LAN
You will need to prepare your systems as follows:

1. have a shared storage area such as NFS mount.
2. have all the computers mount to the same NFS share.

The next step is to install a parallel virtual machine. Check out the project PVM at (Parallel Virtual Machine).
You may want to set up ssh on all the machines to make PVM run better and more secure.

Go to CPAN and fetch the module Parallel::PVM, which is the Perl interface to the parallel virtual machine.

PVM has pretty much everything you want for parallel computing, except for, distributed shared memory (which you really don't need anyway). Your code can be written in manager-worker mode. The manager will launch workers, and the launching process is transparent to your program. PVM will decide which machine the program will run on, depending on the work load.

By the way, the machines do not have to be of the same architecture or running the same OS. It's perfectly ok to run a cluster of linux/solaris/windows mixed nodes.

I am not going to say more here, I hope that you find this information useful, and take joy in doing research into PVM.


Comment on Re: distributed computing
Select or Download Code
Re^2: distributed computing
by Win (Novice) on Sep 02, 2005 at 08:19 UTC
    I've been thinking about this problem again and I think that I am making it much more complex than it needs to be.

    Each PC that is available to perform an operation with a flat file could dip in at intervals into a single folder (on a network) and then remove the longest present flat file (request file) to the PC in question, so preventing other PCs using it. The process of grabbing only takes place when no other flat file is being processed on that machine. The file taken would be the one that has been in the holding folder for the longest period. The time intervals that each PC attempts to grab a file could be determined by CPU activity, and this would determine the probability of each machine taking on the task. The time intervals in which the holding folder is checked could be set so that it is impossible for two of the PCs to attempt to take a file at the same time, although I have no idea how I would work that out. Can anyone help me with that? 2 or more PCs taking a file at the same time is the only problem I can see with this.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://488163]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (15)
As of 2014-09-02 15:37 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (25 votes), past polls