Re: Parallel::Runner and Amazon SQS Issue

[runrig]

Seeing as how this is probably a one-shot, I might just select data in the main loop, and feed it to Amazon in parallel processes. It shouldn't take ALL that long to select 70G of data. The bottleneck, I would think, would be the Amazon request. Something like:

while ( my $records = fetch_next_5000_records_from_sth() ) {
  $runner->run( ..feed records to Amazon... );
)

Update: And it looks like you really only feed one record at a time to the web service, so it might as well be:

while ( my $record = $sth->fetch() ) {
  $runner->run(... feed record to service ...);
}

[stonecolddevin]

Hmmm good point. I've implemented this and unless something explodes, all looks like it's going according to plan. Thanks runrig!