Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re: Parallelization of multiple nested loops

by BrowserUk (Pope)
on Feb 07, 2018 at 13:44 UTC ( #1208634=note: print w/replies, xml ) Need Help??


in reply to Parallelization of multiple nested loops

Starting a new process to construct a single string and then pass it back to the parent (and its parent and its parent...) for accumulation is crazy -- and as you know, slow.

This code produces the same data into a file in less than 1 hour:

#! perl -slw use strict; use Time::HiRes qw[ time ]; use Algorithm::Combinatorics qw[ variations_with_repetition ]; my $start = time; my @in = qw[ 0.0 0.2 0.4 0.6 0.8 1.0 ]; my $iter = variations_with_repetition( \@in, 11 ); my $c = 0; while( $_ = $iter->next ) { print join "\t", @$_, 1 , 1; ++$c; } printf STDERR "Took: %f seconds [$c]\n", time() - $start; __END__ C:\test>1208610 | wc -l Took: 3444.838110 seconds [362797056] 362797056

And you can read it from that file, one line at a time to avoid blowing your memory, in about 5 minutes.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
In the absence of evidence, opinion is indistinguishable from prejudice. Suck that fhit

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1208634]
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (9)
As of 2018-02-20 09:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    When it is dark outside I am happiest to see ...














    Results (268 votes). Check out past polls.

    Notices?