http://www.perlmonks.org?node_id=1069452


in reply to Re^7: Threaded Code Not Faster Than Non-Threaded -- Why?
in thread Threaded Code Not Faster Than Non-Threaded -- Why?

It isn't nearly finished and is undergoing a transformation since I typed my response, but at the time of this writing you can find the threading code here

One of the first things you'll notice is that I didn't implement a "guard" or "doorman". I'm just stuffing everything in the queue for now. Even on directory trees larger than 200GB it barely takes more than 200MB RAM.

Tommy
A mistake can be valuable or costly, depending on how faithfully you pursue correction
  • Comment on Re^8: Threaded Code Not Faster Than Non-Threaded -- Why?

Replies are listed 'Best First'.
Re^9: Threaded Code Not Faster Than Non-Threaded -- Why?
by BrowserUk (Patriarch) on Jan 06, 2014 at 07:06 UTC

    I'd suggest trying the following modification:

    my $thread_term   :shared = 0;
my $threads_init  :shared = 0;
my $file_read_sem : shared;                                           
+                   ### Add this

...

   WORKER: while ( !$thread_term && defined ( my $file = $work_queue->
+dequeue ) )
   {
     {                                                                
+                   ### Add this
         lock $file_read_sem;                                         
+                   ### Add this
         open my $fh, '<', $file or do { lock $d_counter; $d_counter++
+; next WORKER };

         my $data = <$fh>;

         close $fh;
      }                                                               
+                   ### Add this

    [download]
    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
      I'd suggest trying the following modification

      ...To prevent disk thrashing?

      Tommy
      A mistake can be valuable or costly, depending on how faithfully you pursue correction
[reply]
        To prevent disk thrashing?

        Yes. But you'll need to try it; and the effects will probably vary for different file systems.

        For lots of small files it might speed things up; for lots of big ones it might slow things down. For a mix; you'll need to measure.

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom