Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re^2: Grid computing & Amazon EC2

by Krambambuli (Deacon)
on Jan 14, 2009 at 07:51 UTC ( #736152=note: print w/ replies, xml ) Need Help??


in reply to Re: Grid computing & Amazon EC2
in thread Grid computing & Amazon EC2

I want to split the burden of processing a huge file of data over a number of instances. Basically, split the starting file in parts, give each instance a part to work upon, then collect and (re-)assemble the result data when it becomes available. The parts can be processed independently one of each other, it's basically a validation.

I have something similar working fine with forks - but I'd want to check how it does when splitting the job over a number of different machines.

The take-that/give me what you've done part is where I'm looking for solutions, as I'm unsure about how to communicate with the (remote) processes that do there work on the launched instances.

Thanks.


Krambambuli
---


Comment on Re^2: Grid computing & Amazon EC2
Re^3: Grid computing & Amazon EC2
by andye (Curate) on Jan 15, 2009 at 11:58 UTC
    The take-that/give me what you've done part is where I'm looking for solutions, as I'm unsure about how to communicate with the (remote) processes that do there work on the launched instances.

    I've used Amazon's own messaging service for this with success - that's its intended use AFAIR.

    HTH!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://736152]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (13)
As of 2014-07-28 18:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (206 votes), past polls