Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Parallel Modules ?

by Gary Yang (Acolyte)
on Nov 20, 2011 at 09:03 UTC ( #939047=perlquestion: print w/ replies, xml ) Need Help??
Gary Yang has asked for the wisdom of the Perl Monks concerning the following question:

Hi BrowserUk, use Async is a good solution. Since return value from paralleled subs is hash reference, I use the AsyncData use Async; $proc = AsyncData->new(sub {...}); But, I got errors. It says, "Magic number checking on storable string failed at /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Storable.pm line 416, at /usr/lib/perl5/site_perl/5.8.8/Async.pm line 128". I google searched. But, did not find any good explanation and solution. Do you have any idea? Gary

Comment on Parallel Modules ?
Re: Parallel Modules ?
by Corion (Pope) on Nov 20, 2011 at 09:10 UTC

    You don't tell us which modules you looked at. I would look at threads.

Re: Parallel Modules ?
by Anonymous Monk on Nov 20, 2011 at 09:16 UTC
Re: Parallel Modules ?
by BrowserUk (Pope) on Nov 20, 2011 at 09:28 UTC
    Any parallel module can handle this case?

    Yes, threads. Ostensibly it should be as simple as:

    use threads; use Your::Module; my @AoH = map $_->join, map{ async( \&{ "Your::Module::$_" } ) } qw[ sub1 sub2 sub3 sub4 sub5 sub5 sub7 sub8 sub9 sub10 ]; ## do something with the 10 hashrefs here

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      Hi BrowserUk,

      use Async is a good solution. The return value from paralleled subs is hash reference, I use the AsyncData

      use Async;
      $proc = AsyncData->new(sub {...});

      But, I got errors. It says, "Magic number checking on storable string failed at /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Storable.pm line 416, at /usr/lib/perl5/site_perl/5.8.8/Async.pm line 128". I google searched. But, did not find any good explanation and solution. Do you have any idea?

      Gary

        use Async is a good solution.

        Err, no. Use Async is a terrible solution. look at the errors it is giving you.

        I suggested use threads, which would be a good solution. For a start you wouldn't be having the errors you now have,


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Parallel Modules ?
by sundialsvc4 (Abbot) on Nov 20, 2011 at 13:36 UTC

    All right ... where do these subroutines fetch their data from?   Have you thoroughly established that every component of that entire system possesses the capability to deliver 10 times as much data per second as they presently do?

    Have you taken a murderously-close look at the algorithm, to verify beyond all reproach that, each time they execute, they are fetching precisely what they must fetch and nothing more than is needed to solve the problem, and that they are never fetching anything that they have already requested before?

    I am cordially saying that because, way too often, I have watched efforts to “parallelize” something produce a final product that is noticeably(!) slower than its predecessor.   Very fine parallelization engines do of course exist and several of these have already been cited.   But I suggest that you cross-examine your algorithm first.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://939047]
Approved by BrowserUk
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (7)
As of 2014-12-21 18:49 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (106 votes), past polls