Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Sharing statics variable between Threads

by rodeo (Initiate)
on Aug 30, 2013 at 09:07 UTC ( #1051579=perlquestion: print w/ replies, xml ) Need Help??
rodeo has asked for the wisdom of the Perl Monks concerning the following question:

Hello everybody, I have a problem, where I didn't find some solutions until now. I have a program for calculating. For it I must load large data (10GB). This files will be written in a structure. After that I call a function which calculate with this data and an input (load by a excel file). This funtion have 5 interlaced for-loops. The results are well and all runs fine but to get a result I must wait to 10 hours. I have found that I can do the calculates parallel with the Parallel::ForkManager, where the steps of the loop will be swap out at a extra core. The first tests was fine but the used RAM will be multiply by the number of cores. With the modules threads/threads::shared you can share hashes and arrays. For sharing the loaded data, i changed the structure to a hash. Unfortunately the calculating needs more than 24 hours so I canceled the programm. The same happens when I use onlny 2 cores. Now my question: it's there a possibility to share static variables (in the loops the data of the loaded data don't will be changem, I only reading from the structure). Or somebody have an other idea how I can release this problem? Thank you very much
my $maxProc=4; my $pm = new Parallel::ForkManager($maxProc); for(my $idy=0;$idy<=$#ySurf;$idy++) { my $currY = $ySurf[$idy]; for(my $alfaIdx=0;$alfaIdx<=$#alfas;$alfaIdx++) { $pm->start and next; my $alfa = $alfas[$alfaIdx]; my $norVectX = cos($alfa); my $norVectY = 0; my $norVectZ = sin($alfa); my $norTransf=TransfNorVec($input->{az}, $input->{el}, $norVec +tX, $norVectY, $norVectZ, $Side); my $intSurfx = $input->{MiddleX} - $input->{r}*$norVectX; my $intSurfy = $currY; my $intSurfz = ($input->{MiddleZ}+$input->{zOffset})-$input->{ +r}*$norVectZ; my $center = Divide($input->{w}, $input->{h}, $input->{M}, $in +put->{N}, $intSurfx, $intSurfy, $intSurfz, $alfa); my $surf=0; for(my $idx=0;$idx<$center->{count};$idx++) { my $xT = $center->{Px}[$idx]; my $yT = $center->{Py}[$idx]; my $zT = $center->{Pz}[$idx]; my $xTr = coordSystTransf($input->{az}, $input->{el}, $inp +ut->{MiddleX}, $input->{MiddleY}, $input->{MiddleZ}, $input->{ZeroX}, + $input->{ZeroY}, $input->{ZeroZ}, $xT, $yT, $zT, $Side); my $sqDiag = ($input->{w}/$input->{M})**2+($input->{h}/$in +put->{N})**2; my $P = new Points(); $P->{count}=0; for(my $dataIdx=0;$dataIdx<$data->{count};$dataIdx++) { my $inSphere = Check($sqDiag, $xTr->{xe}, $xTr->{ye}, +$xTr->{ze}, $data->{abs_x}[$dataIdx], $data->{abs_y}[$dataIdx], $data +->{abs_z}[$dataIdx]); if($inSphere == 1) { $P->{Px}[$P->{count}]=($data->{magX}[$dataIdx])*co +s($data->{phX}[$dataIdx]); $P->{Py}[$P->{count}]=($data->{magY}[$dataIdx])*co +s($data->{phY}[$dataIdx]); $P->{Pz}[$P->{count}]=($data->{magZ}[$dataIdx])*co +s($data->{phZ}[$dataIdx]); $P->{count}++; } } my $Poynt = new Trans(); if(defined($P->{Px}[0]) && defined($P->{Py}[0]) && defined +($P->{Pz}[0])) { $Poynt->{xe} = summe($P->{Px})/$P->{count}; $Poynt->{ye} = summe($P->{Py})/$P->{count}; $Poynt->{ze} = summe($P->{Pz})/$P->{count}; my $innerProd = $Poynt->{xe}*$norTransf->{xe}+$Poynt-> +{ye}*$norTransf->{ye}+$Poynt->{ze}*$norTransf->{ze}; $surf += ((($input->{w})/$input->{M})*(($input->{h})/$ +input->{N}))*$innerProd; } } print "\ny= ".$currY."; alfa= ".$alfa .";\n"; if($surf == 0) { print 1e-100 ."dB\n"; }else{ print sprintf("%.4f", 20*log(abs($surf))) ." dB\n"; } $pm->finish; } $pm->wait_all_children; }

Comment on Sharing statics variable between Threads
Download Code
Re: Sharing statics variable between Threads
by BrowserUk (Pope) on Aug 30, 2013 at 11:06 UTC
    Or somebody have an other idea how I can release this problem?

    Post the loop code.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Sharing statics variable between Threads
by roboticus (Canon) on Aug 30, 2013 at 11:54 UTC

    rodeo:

    When I'm trying to make code faster, threading isn't the first tool I reach for. Usually you'll have the best results if you improve your algorithm(s). You don't show your code or algorithm, so I can't offer any suggestions there. However, you mention that you have five interlaced for loops. I think I'd start looking at those loops to see if you can tune things up there.

    Verify that you're using data structures suited to your task. If you constantly find yourself looping through an array to find a data element, perhaps it would be better to use a hash so you can access data by name. Frequently that can remove some loops from your code, and can be a dramatic speed increase.

    Once I'm happy with my algorithm and data structures, the next thing I try when I need to make something faster is to tune up the algorithm I'm currently using: First, I move calculations out of the most frequently executed loop(s) when possible. In the example below (first section), rather than compute sin($j) 100*100*100 times, we compute it only 100 times. Similarly we compute cos($k) only 100*100 times. The overall savings can add up quite a bit!

    Next, if you have some expensive subroutines that you call frequently with the same parameters, you might find Memoize handy: It will store the results of a call and reuse the result when you call it again with the same value(s). As the second example shows, it can give a big win for only two lines of code.

    There are many other optimization tricks you can use, too.

    Now I must confess: I've never used perls code profiler, though I've been meaning to get around to it. But find and use a good profiler to find out what your code is doing that takes the most time. That will show you where to focus your attention. If you use a profiler rather than guessing, it can point you to some surprising places. (Most of my speed-critical code is in C/C++, where I *do* profile my code.) You might, for example, find that half your execution time is spent formatting your data for printing your progress to a log. In that case, you might log less information and save a lot of time. (As an example, see how loop3 is *so* much slower than loop1?)

    Usually I don't reach for threading/forking until I've used the above techniques. If the problem is easy to split into independent pieces, then threading can be easy enough and give you some good results. But threading can often be difficult to apply to some problems. If you split your problem incorrectly, then you'll find that your program takes *longer* to run. If you need assistance with threading, you'll want to appeal to a monk who uses threading more than me (e.g., BrowserUk).

    $ perl 1051579.pl s/iter loop3 loop1 loop2 loop3 17.5 -- -88% -92% loop1 2.08 740% -- -36% loop2 1.33 1210% 56% -- no memoize: 30.9833 wallclock secs (30.98 usr + 0.00 sys = 30.98 CP +U) @ 1613.94/s (n=50000) with memoize: 0.816253 wallclock secs ( 0.82 usr + 0.00 sys = 0.82 C +PU) @ 60975.61/s (n=50000) $ cat 1051579.pl #!/usr/bin/perl use strict; use warnings; use Benchmark qw(:all :hireswallclock); #use Memoize; my ($cnt1, $cnt2); sub loop1 { $cnt1=0; for my $j (0 .. 100) { for my $k (0 .. 100) { for my $l (0 .. 100) { $cnt1 += sin($j) + cos($k) + sqrt($l); } } } } sub loop2 { $cnt2=0; for my $j (0 .. 100) { my $sin_j = sin($j); for my $k (0 .. 100) { my $cos_k = cos($k); for my $l (0 .. 100) { $cnt2 += $sin_j + $cos_k + sqrt($l); } } } } open my $FH, '>', '/dev/null'; sub loop3 { $cnt1=0; for my $j (0 .. 100) { for my $k (0 .. 100) { for my $l (0 .. 100) { $cnt1 += sin($j) + cos($k) + sqrt($l); printf $FH "% 3u % 3u % 3u : %f\n", $j, $k, $l, $cnt1; } } } } cmpthese 5, { loop1 => q[ loop1( ) ], loop2 => q[ loop2( ) ], loop3 => q[ loop3( ) ], }; # double-check that the routines do the same thing! print "**** $cnt1 != $cnt2 ****\n" if abs($cnt1-$cnt2)>.0001; print "\n\n"; no warnings 'recursion'; # hideously expensive subroutine sub glarf { my $t = shift; return 1 if $t <2; return 2 if $t <3; return glarf($t-1)+glarf($t-2)+glarf($t-3); } my $iter=50000; my $fibsub = sub { glarf(10); #glarf(20); #glarf(30); glarf(40); glarf(50); glarf(100 +); }; timethis($iter, $fibsub, 'no memoize' ); # These two lines of code make it *much* faster use Memoize; memoize('glarf'); timethis($iter, $fibsub, 'with memoize' );

    Update: It seems rodeo has posted the code while I was writing this. But I'm stopping for now because I've got to get to work...

    ...roboticus

    When your only tool is a hammer, all problems look like your thumb.

Re: Sharing statics variable between Threads
by BrowserUk (Pope) on Aug 30, 2013 at 12:30 UTC

    Ignore this.

Re: Sharing statics variable between Threads
by BrowserUk (Pope) on Aug 30, 2013 at 13:32 UTC

    How big are:

    1. @ySurf
    2. @alfas
    3. $data->{count}

    How big, or what range of sizes does $center->{count} take?


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      The sizes of @ysurf and @alfas are round about 10...15. $data->{count} more than 50.000.000. The count of center is thousands and more

        Where does $data come from?


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        Your code is full of silly inefficiencies; like these 3 calculations:

        $data->{magX}[ $dataIdx ] * cos( $data->{phX}[ $dataIdx ] ) $data->{magY}[ $dataIdx ] * cos( $data->{phY}[ $dataIdx ] ) $data->{magZ}[ $dataIdx ] * cos( $data->{phZ}[ $dataIdx ] )

        being performed for all 50 million $data points; and all 50 million being repeated for every one of the "thousands" of $center points. That's 3 * "thousands" * "50.000.000" of redundant, repeated calculations.

        Also, whilst I won't pretend to fully understand your code in isolation of what TransfNorVec(), coordSystTransf() & Check() subs do, it seems fairly clear you are checking a bunch of 3D points against another bunch of 3D points and accumulating statistics about how many of the former are within some distance of some of the latter.

        And you are going about it by brute force.

        With a clearer picture of the data (in $data (what an awful name) and $center), it ought to be possible to order (sort or structure -- perhaps 3-dimensionally) them, so that for any given $input only a small subset of the 'thousands by millions' of comparisons (and associated calculations) need be done.

        For example, if the point's X value is 50 and the radius of the compare is 5, there is no point in comparing against any other point who's X-value is outside of 45 thru 55. Sub-setting the larger range early could save billions of calculations, comparisons, object constructions and function calls.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1051579]
Approved by hdb
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (18)
As of 2014-09-18 14:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (116 votes), past polls