Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re^5: Threaded Code Not Faster Than Non-Threaded -- Why?

by BrowserUk (Pope)
on Jan 05, 2014 at 21:16 UTC ( #1069421=note: print w/ replies, xml ) Need Help??


in reply to Re^4: Threaded Code Not Faster Than Non-Threaded -- Why?
in thread Threaded Code Not Faster Than Non-Threaded -- Why?

It seems like it will take a bit of experimentation to figure out how long to sleep and how deep to keep the queue.

These are not as critical as you might first think.

  1. sleep: A thread that calls $Q->pending once every millisecond will consume so few cycles that it barely registers on taskmanager/top.

    Try this:

    perl -MThread::Queue -E"$Q=new Thread::Queue; $Q->pending while Win32: +:Sleep( 1 )"

    On my system, that shows up in Process Manager as 0.0%.

    So, running it at a frequency of once every timeslice -- ~10 milliseconds -- will consume negligible cpu whilst ensuring that the queue is (re)populated in a timely manner.

  2. Queue Length: A queue, shared between 16 threads, populated with a million longish paths, will consume less than 200MB.

    Try:

    perl -Mthreads -MThread::Queue -E"async{ sleep }->detach for 1 .. 16; +$Q=new Thread::Queue; $Q->enqueue( qq[/a/longish/path/and/filename$_] + ) for 1 .. 1e6; sleep;"

    Which given a typical modern system with 4-16 GB is nothing to concern ourselves with; and is at least 3 order of magnitude greater than I'd recommend as a starting point (100 for 16 threads).

For IO-bound process like this, tuning either or both variables over quiet a large range of values will have little or no effect on overall throughput.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.


Comment on Re^5: Threaded Code Not Faster Than Non-Threaded -- Why?
Select or Download Code
Re^6: Threaded Code Not Faster Than Non-Threaded -- Why?
by Tommy (Chaplain) on Jan 06, 2014 at 03:15 UTC

    The results are in. I forked and refactored the code to follow the pattern you laid out (BrowserUk). The threaded part of the code is over 2X faster now! Thanks again!

    Tommy
    A mistake can be valuable or costly, depending on how faithfully you pursue correction

      Could you post (a link to) your latest code?


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

        It isn't nearly finished and is undergoing a transformation since I typed my response, but at the time of this writing you can find the threading code here

        One of the first things you'll notice is that I didn't implement a "guard" or "doorman". I'm just stuffing everything in the queue for now. Even on directory trees larger than 200GB it barely takes more than 200MB RAM.

        Tommy
        A mistake can be valuable or costly, depending on how faithfully you pursue correction

        It isn't nearly finished and is undergoing a transformation since I typed my response, but at the time of this writing you can find the threading code here

        One of the first things you'll notice is that I didn't implement a "guard" or "doorman". I'm just stuffing everything in the queue for now. Even on directory trees larger than 200GB it barely takes more than 200MB RAM.

        Tommy
        A mistake can be valuable or costly, depending on how faithfully you pursue correction

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1069421]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (15)
As of 2014-10-01 17:20 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    What is your favourite meta-syntactic variable name?














    Results (30 votes), past polls