Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re: Parallelization of multiple nested loops

by BrowserUk (Pope)
on Feb 07, 2018 at 13:44 UTC ( #1208634=note: print w/replies, xml ) Need Help??


in reply to Parallelization of multiple nested loops

Starting a new process to construct a single string and then pass it back to the parent (and its parent and its parent...) for accumulation is crazy -- and as you know, slow.

This code produces the same data into a file in less than 1 hour:

#! perl -slw use strict; use Time::HiRes qw[ time ]; use Algorithm::Combinatorics qw[ variations_with_repetition ]; my $start = time; my @in = qw[ 0.0 0.2 0.4 0.6 0.8 1.0 ]; my $iter = variations_with_repetition( \@in, 11 ); my $c = 0; while( $_ = $iter->next ) { print join "\t", @$_, 1 , 1; ++$c; } printf STDERR "Took: %f seconds [$c]\n", time() - $start; __END__ C:\test>1208610 | wc -l Took: 3444.838110 seconds [362797056] 362797056

And you can read it from that file, one line at a time to avoid blowing your memory, in about 5 minutes.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
In the absence of evidence, opinion is indistinguishable from prejudice. Suck that fhit

Replies are listed 'Best First'.
Re^2: Parallelization of multiple nested loops
by biosub (Initiate) on Feb 20, 2018 at 13:45 UTC
    This works like a charm! 20 min on my computer :) Thanks a LOT

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1208634]
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (2)
As of 2018-07-19 21:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    It has been suggested to rename Perl 6 in order to boost its marketing potential. Which name would you prefer?















    Results (420 votes). Check out past polls.

    Notices?