http://www.perlmonks.org?node_id=1069447


in reply to Re^6: Threaded Code Not Faster Than Non-Threaded -- Why?
in thread Threaded Code Not Faster Than Non-Threaded -- Why?

Could you post (a link to) your latest code?

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
  • Comment on Re^7: Threaded Code Not Faster Than Non-Threaded -- Why?

Replies are listed 'Best First'.
Re^8: Threaded Code Not Faster Than Non-Threaded -- Why?
by Tommy (Chaplain) on Jan 06, 2014 at 04:55 UTC

    It isn't nearly finished and is undergoing a transformation since I typed my response, but at the time of this writing you can find the threading code here

    One of the first things you'll notice is that I didn't implement a "guard" or "doorman". I'm just stuffing everything in the queue for now. Even on directory trees larger than 200GB it barely takes more than 200MB RAM.

    Tommy
    A mistake can be valuable or costly, depending on how faithfully you pursue correction
[reply]

      I'd suggest trying the following modification:

      my $thread_term   :shared = 0;
my $threads_init  :shared = 0;
my $file_read_sem : shared;                                           
+                   ### Add this

...

   WORKER: while ( !$thread_term && defined ( my $file = $work_queue->
+dequeue ) )
   {
     {                                                                
+                   ### Add this
         lock $file_read_sem;                                         
+                   ### Add this
         open my $fh, '<', $file or do { lock $d_counter; $d_counter++
+; next WORKER };

         my $data = <$fh>;

         close $fh;
      }                                                               
+                   ### Add this

      [download]
      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
        I'd suggest trying the following modification

        ...To prevent disk thrashing?

        Tommy
        A mistake can be valuable or costly, depending on how faithfully you pursue correction
[reply]
Re^8: Threaded Code Not Faster Than Non-Threaded -- Why?
by Tommy (Chaplain) on Jan 06, 2014 at 04:55 UTC

    It isn't nearly finished and is undergoing a transformation since I typed my response, but at the time of this writing you can find the threading code here

    One of the first things you'll notice is that I didn't implement a "guard" or "doorman". I'm just stuffing everything in the queue for now. Even on directory trees larger than 200GB it barely takes more than 200MB RAM.

    Tommy
    A mistake can be valuable or costly, depending on how faithfully you pursue correction
[reply]

In Section Seekers of Perl Wisdom