Re^2: Grid computing & Amazon EC2

[Krambambuli]

I want to split the burden of processing a huge file of data over a number of instances. Basically, split the starting file in parts, give each instance a part to work upon, then collect and (re-)assemble the result data when it becomes available. The parts can be processed independently one of each other, it's basically a validation.

I have something similar working fine with forks - but I'd want to check how it does when splitting the job over a number of different machines.

The take-that/give me what you've done part is where I'm looking for solutions, as I'm unsure about how to communicate with the (remote) processes that do there work on the launched instances.

Thanks.

[andye]

The take-that/give me what you've done part is where I'm looking for solutions, as I'm unsure about how to communicate with the (remote) processes that do there work on the launched instances.

I've used Amazon's own messaging service for this with success - that's its intended use AFAIR.

HTH!