Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Threads From Hell #1: How To Share A Hash [SOLVED]

by karlgoethebier (Abbot)
on May 13, 2015 at 19:07 UTC ( [id://1126584]=perlquestion: print w/replies, xml ) Need Help??

karlgoethebier has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,

i figured this out by RTFM:

#!/usr/bin/env perl use strict; use warnings; use threads; use threads::shared; use Thread::Semaphore; use Math::BigInt; use Data::Dump; use feature qw(say); my @numbers = ( 1 .. 10 ); my %result : shared; my $semaphore = Thread::Semaphore->new(4); my @threads = map { $semaphore->down; threads->create( \&process, $_ ); } @numbers; $_->join for @threads; dd \%result; sub process { my $number = shift; my $factorial = factorial($number); say qq($number $factorial); eval { $result{$number} = $factorial; }; say $@ if $@; $semaphore->up; } sub factorial { my $number = shift; Math::BigInt->bfac($number); } __END__ karls-mac-mini:monks karl $ ./semaphore.pl ... 10 3628800 Invalid value for shared scalar at ./semaphore.pl line 29. ...

It works, but the only thing i couldn't manage is to assign the results to a hash.

I did some more RTFM about how to accomplish this but without success.

What do i miss?

Thank you very much for any hint and best regards,

Karl

P.S.: I asked something similar at noon it the CB and posted an example in my scratchpad. Corion kindly advised my not to use Thread::Semaphore and pointed me to the canonical advice given by BrowserUK which uses Thread::Queue. But i couldn't resist to use Thread::Semaphore

«The Crux of the Biscuit is the Apostrophe»

Replies are listed 'Best First'.
Re: Threads From Hell #1: How To Share A Hash
by BrowserUk (Patriarch) on May 13, 2015 at 20:00 UTC

    You're trying to assign a Math::BigInt object to a shared hash -- hence the "Invalid value for shared scalar" msg.

    You can however assign the value of that object:

    use strict; use threads; use threads::shared; use Math::BigInt; use Data::Dump qw[ pp ]; use feature qw(say); my @numbers = ( 1 .. 10 ); my %result : shared; my @threads = map { threads->create( \&process, $_ ); } @numbers; $_->join for @threads; for my $key ( sort{ $a <=> $b } keys %result ) { say "$key : $result{ $key }"; } sub process { my $number = shift; my $factorial = factorial($number); lock %result; $result{$number} = $factorial->bstr; ### Extract the value from th +e object } sub factorial { my $number = shift; Math::BigInt->bfac($number); }

    Produces:

    C:\test>junk 1 : 1 2 : 2 3 : 6 4 : 24 5 : 120 6 : 720 7 : 5040 8 : 40320 9 : 362880 10 : 3628800
    Corion kindly advised my not to use Thread::Semaphore ... But i couldn't resist to use Thread::Semaphore

    Dunno quite what you thought you were achieving with it, but whatever it was, it wasn't doing anything useful.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
    In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
      "...whatever it was, it wasn't doing anything useful."

      Well, i would say it depends.

      It was for learning purposes, a kind of finger exercise.

      I didn't mention that i don't have any experience using threads in Perl. My first approach.

      I always felt like i'll miss something if i don't go over that next bridge.

      My little effort was also inspired by my last thread about P::FM, because that stuff doesn't work on windows. I am sorry that i can't show the error messages, because i don't have a windows box with Perl installed at my disposal tonight.

      Also pure curiosity played a role. I have a basic understanding how a counting semaphore works but this stuff is still a kind of magic for me.

      And the code i figured out looks very similar to my forking example and doesn't perform so bad.

      That's why i couldn't resist.

      My very best regards, Karl

      «The Crux of the Biscuit is the Apostrophe»

        And the code i figured out looks very similar to my forking example and doesn't perform so bad.

        So... you were trying to restrict the number of concurrent threads to 4? Yes. It works for that purpose.

        (I originally thought is was some attempt to prevent concurrent access to the shared hash; ie. a substitute for lock. My mistake.)


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
        In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
      "...assign a Math::BigInt object...assign the value of that object...

      Do'h! That hurts.

      In German it (my behavior) is called "Den Wald vor lauter Bäumen nicht sehen" which should translate to "to miss the forest for the trees" - i think.

      Update: Minor change in wording to avoid misunderstanding.

      Thank you very much for advice and my best regards, Karl

      «The Crux of the Biscuit is the Apostrophe»

Re: Threads From Hell #1: How To Share A Hash
by BrowserUk (Patriarch) on May 13, 2015 at 20:05 UTC

    Alternatively, if you want/need to keep the Math::BigInt object intact, you can use threads::shared::shared_clone:

    use strict; use threads; use threads::shared; use Math::BigInt; use Data::Dump qw[ pp ]; use feature qw(say); my @numbers = ( 1 .. 10 ); my %result : shared; my @threads = map { threads->create( \&process, $_ ); } @numbers; $_->join for @threads; for my $key ( sort{ $a <=> $b } keys %result ) { say "$key : ", $result{ $key }, ' => ', $result{ $key }->bstr; } sub process { my $number = shift; my $factorial = factorial($number); lock %result; $result{$number} = shared_clone( $factorial ); ## clone the object } sub factorial { my $number = shift; Math::BigInt->bfac($number); }

    Produces:

    C:\test>junk 1 : Math::BigInt=HASH(0x3f5d160) => 1 2 : Math::BigInt=HASH(0x3f5d190) => 2 3 : Math::BigInt=HASH(0x3f5d148) => 6 4 : Math::BigInt=HASH(0x3f5d178) => 24 5 : Math::BigInt=HASH(0x3f5d118) => 120 6 : Math::BigInt=HASH(0x3f5d130) => 720 7 : Math::BigInt=HASH(0x3f5d160) => 5040 8 : Math::BigInt=HASH(0x3f5d190) => 40320 9 : Math::BigInt=HASH(0x3f5d148) => 362880 10 : Math::BigInt=HASH(0x3f5d178) => 3628800

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
    In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

      Thanks again BrowserUK.

      But must i really say lock %result?

      Without it the calculation for 2000 numbers is about two times faster twice as fast.

      Edit: Minor change of wording.

      Best regards, Karl

      «The Crux of the Biscuit is the Apostrophe»

        But must i really say lock %result?

        If you can guarantee no duplicates in your list of numbers; then no.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
        In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
Re: Threads From Hell #1: How To Share A Hash [SOLVED]
by BrowserUk (Patriarch) on May 15, 2015 at 18:00 UTC
    Corion kindly advised my not to use Thread::Semaphore and pointed me to ... Thread::Queue.

    I finally got around to putting together proof of the wisdom of Corion's advice.

    This is a version of your Thread::Semaphore code, limiting to 4 concurrent threads and calculating factorials 1000! .. 2000!:

    Total runtime: Took 47.511607 seconds

    This version does the same calculations using the same number of concurrent threads, but ditches Thread::Semaphore in favour of Thread::Queue to queue the 1000 numbers to 4 reused threads, thus saving the startup and teardown costs of 996 threads:

    Total runtime: Took 31.290966 seconds; giving a 33% saving of time.

    But the biggest lesson of threading, is when not to use it. This version ditches threads altogether and uses the obvious optimisation:

    Total runtime: Took 0.685503 seconds. Same results, but a 98.5% time saving over the first version above.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
    In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
      "But the biggest lesson...the obvious optimisation..."
      "Du sprichst ein großes Wort gelassen aus." (Johann Wolfgang von Goethe)

      Very impressive and instructional. What else should i say?

      Update:

      There are two little things i would like to add:

      My basic theme wasn't to calculate factorials using threads.

      I wanted to explore multithreading and thought the best idea would be to use some "expensive" calculation for this. Like in a classic homework.

      Unfortunately you provided a much better algorithm that avoids multithreading ;-)

      I fear i hit a basic problem in this context.

      And apropos little quirks:

      I wrote:

      sub process { my $number = shift; # lock %result; $result{ threads->tid() } = shared_clone( { $number => factorial($number) } ); $semaphore->up; }

      It seems like i really need to say something like:

      for my $key ( sort { $a <=> $b } keys %result ) { my $ref = $result{$key}; while ( my ( $k, $v ) = each %{$ref} ) { say qq($k => $v); } }

      ...to iterate over the results. From the docs:

      "each HASH does not work properly on shared references embedded in shared structures"

      Thanks for this advice and my best regards, Karl

      «The Crux of the Biscuit is the Apostrophe»

      I'm struggling with this:

      $Q->enqueue( 1000..2000, (undef) x 4 );

      The docs say:

      ->enqueue(LIST) Adds a list of items onto the end of the queue.

      I dumped it:

      karls-mac-mini:monks karl$ ./queue.pl bless({ # tied threads::shared::tie queue => [ # tied threads::shared::tie 1000 .. 2000, undef, undef, undef, undef, ], }, "Thread::Queue") Took 13.820966 seconds

      What are the four undefs good for?

      Best regards, Karl

      P.S.: But my box is faster than yours ;-)

      «The Crux of the Biscuit is the Apostrophe»

        What are the four undefs good for?

        They cause these loops to terminate:

        while( my $number = $Q->dequeue ) {
        P.S.: But my box is faster than yours ;-)

        No surprise there. My Q6600 is over 7 years old.

        Mind you, I make better use of it than most people do of their Haswells :)


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
        In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
Re: Threads From Hell #1: How To Share A Hash [SOLVED]
by marioroy (Prior) on Aug 29, 2015 at 04:49 UTC

    I had always wanted to share a solution using MCE which comes with a sequence generator. This allows one to use a pool of workers seamlessly while iterating through a sequence of numbers.

    The first example is a modified version of the demonstration by BrowserUk.

    use strict; use threads; use threads::shared; use MCE::Flow; use Math::BigInt; use feature qw(say); my %result : shared; mce_flow_s { chunk_size => 1, max_workers => 4 }, \&process, 1, 1000; for my $key ( sort{ $a <=> $b } keys %result ) { say "$key => ", $result{ $key }->bstr; } sub process { my $number = $_; my $factorial = factorial($number); $result{$number} = shared_clone( $factorial ); ## clone the object } sub factorial { my $number = shift; Math::BigInt->bfac($number); }

    The next example requires the upcoming MCE 1.7 release, currently residing in GitHub. Notice the use of MCE::Shared and the capital letter S in Shared for the variable attribute.

    use strict; use MCE::Flow; use MCE::Shared; use Math::BigInt; use feature qw(say); my %result : Shared; mce_flow_s { chunk_size => 1, max_workers => 4 }, \&process, 1, 1000; for my $key ( sort{ $a <=> $b } keys %result ) { say "$key => ", $result{ $key }->bstr; } sub process { my $number = $_; my $factorial = factorial($number); $result{$number} = $factorial; # the object is shared automatical +ly } sub factorial { my $number = shift; Math::BigInt->bfac($number); }

    Both complete in the same time frame. Threads is not required to run MCE::Shared as it supports threads and processes alike. Thus, one may include threads if desired.

    Init is another way to specify MCE options.

    MCE::Flow::init { chunk_size => 1, max_workers => 4, }; mce_flow_s \&process, 1, 1000;

    Kind regards, Mario

      Thank you very much marioroy for sharing this! Best regards, Karl

      «The Crux of the Biscuit is the Apostrophe»

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1126584]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others musing on the Monastery: (5)
As of 2024-03-19 09:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found