Re^2: Is it possible to create a single Hash-of-Hash.. with multiple threads..

[gsat]

Thanks much for your time. Your code is simple & works fine - I guess because it is simple 2 level deep & the subroutine is not increasing the depth while working.

The data structure - I am handling with, will increase the structure depth. Basically branch-out, like a growing tree, every thread working on a unique branch is OK.

As other suggestions came like - Storing output in a new structure - and finally merging it - is fine - how to return different sized depth of every branch structure varying between 9 to 14, number of branches would few tens of thousands - and join them to form the final tree.

The starting structure that I need to share across threads - will be having depths like 4-6 level, when threads finished working every branch will have 9-14 levels.

BTW, in my experience, the %hash kind of usage are not very convenient, rather - I prefer reference of hash-ref-of-hash-ref - right from the root. I used to like..

$root->{$branch1}->{$branch2}..

Overall - so far in my understanding - perl threads is not cool, no matter what - It really cannot do big job with complex data structure in shared way - absolutely no references shared between threads. Internally every thread is creating another copy, so with GBs of data-structure I cannot really launch many threads.

My single-thread serialized code is working great though - it can do serious work on massive data-structures. It lacks true parallelism..

Perl Threads implementation so far, seems very inadequate - it is good for primary school student exhibiting how parallel programming works.

I wish next gen threads will be able to do what I am asking - seamless work on the shared structure over GPUs too [why not - being Interpreted, won't stop it running in GPU I guess. :) - anyway Thanks a lot again!!

[BrowserUk]

Perl Threads implementation so far, seems very inadequate - it is good for primary school student exhibiting how parallel programming works.

That is a wrong conclusion drawn through a lack of understanding.

If you would care to describe your application in detail -- not just generics, but rather scales and reasons; with examples of data and processing requirements -- then you would (probably) get help with finding definitive, practical and efficient solutions to those requirements.

Your OP asks a simplistic question with an obvious answer: Yes, of course a multi-level hash can be constructed using multiple threads. It is simple and trivial:

#! perl -slw
use strict;
use threads;
use threads::shared;

sub displayHash {
    my( $ref, $pad ) = @_;
    return "\n" unless keys %{ $ref };
    my $buf = '';
    for my $key ( sort keys %{ $ref } ) {
        $buf .= "$pad" if length $buf;
        $buf .= "{$key}" . displayHash( $ref->{ $key }, $pad . '   ' )
+;
    }
    return $buf;
}

sub worker {
    my( $ref, $reps ) = @_;

    for( 1 .. $reps ) {
        my $copyRef = $ref;
        for my $step ( map chr( 97 + rand 26 ), 1 .. int( rand 10 ) ) 
+{
            if( exists $copyRef->{ $step } ) {
                $copyRef = $copyRef->{ $step };
            }
            else {
                lock %{ $copyRef };
                $copyRef = $copyRef->{ $step } = &share( {} );
            }
        }
    }
}

our $REPS //= 20;
our $THREADS //= 4;

my %HoHos : shared;

my @threads = map threads->create( \&worker, \%HoHos, $REPS ), 1 .. $T
+HREADS;

$_->join for @threads;

print displayHash( \%HoHos, '' );
__END__
C:\test>junk91 -REPS=7
{b}{l}{w}{g}
{d}{t}{k}{f}{g}{g}{g}{i}{j}
{f}{j}{v}
{g}{s}{g}{h}{g}
{i}{f}{j}{r}{m}{v}{u}{v}{b}
{k}{d}{f}{q}{k}
   {z}{t}{m}{h}{e}{i}
{m}{b}{l}
   {l}{u}{p}{t}{d}{h}
   {q}{k}{y}
{n}{f}{u}{v}{z}{z}{l}
{o}{f}
{p}{u}{s}{w}{n}{t}
{q}{h}
   {y}{k}{d}{l}{a}{n}{a}{i}
{r}{q}{t}{g}
{v}{o}{n}{x}{b}{g}{c}
{w}{a}
{y}{a}
   {d}{y}{j}
{z}{a}{p}{t}{i}{r}{t}
   {y}{i}{g}{c}{v}{z}{z}{k}

But, it is also probably not very helpful to your real application.

[gsat]

Dear Monk BrowserUk! Please take a bow!!

Thank You so much - for enlightening me :) Indeed - it was my lack of understanding.

I have found this entire discussion from everybody - very helpful.

-=-=-=-=-=-=-=-

Now, I need more :) It seems - me and my code is tired of doing so many .. if exists.. else share.. kinda additives before I do any structure change - and I think no way to avoid that. Now a couple of Questions here:

Q1. Is there any easier method?

Q2. Is it possible to grow a local tree or branch data-structure, then create a shared_clone - and graft the new tree to s sub-branch of main shared tree?

Q3. I have few subroutines to be called many times - based on data - from inside my thread_worker. So far I am lacking depth in understanding or visualizing - how/when multiple threads work on different set of data, while calling a subroutine - does the code-inside-subroutine-block gets a replicated inside the thread uniquely? - else how the sanity is retained? - sorry if I am asking too basic question - maybe it is the case, I am confused - and not clear enough - where is my confusion too..

Q4. Inside subroutine - do I have to use every variable as local and not-shared?

Thanks a lot! Again & Again!

[BrowserUk]

1. No.

   You can replace:

               if( exists $copyRef->{ $step } ) {
                    $copyRef = $copyRef->{ $step };
                }
                else {
                    lock %{ $copyRef };
                    $copyRef = $copyRef->{ $step } = &share( {} );
                }
   

   With:

               lock %{ $copyRef };
               $copyRef = $copyRef->{ $step } //= &share( {} )
   

   Which is more compact, but not easier nor more efficient.

2. Yes.

   Two possibilities:

   1. You can build a normal (non-shared) subhash and when complete, use threads::shared::shared_clone() to make a shared copy of it and assign a reference to the copy to a key/value pair in the main shared hash:

      ...
      my %subhash;
      ... # populate it
      lock %mainSharedHash;
      $mainSharedHash{ $key } = shared_clone( \%subhash );
      ...
      

   2. You can create a shared subhash and assign a ref to it without needing to copy:

      ...
      my %subhash :shared:
      ... # populate it
      lock %mainSharedHash;
      $mainSharedHash{ $key } = \%subhash;
      

3. I do not understand the question.

   You'll get better answers if you describe your actual application. Ie. Real requirements garner working responses.

4. Again, your question makes no sense.

   In a subroutine (or anywhere else), you can use unshared variables and/or shared variables as required.