Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^2: Use more threads.

by BrowserUk (Pope)
on Apr 28, 2013 at 06:53 UTC ( #1031039=note: print w/ replies, xml ) Need Help??


in reply to Re: Use more threads.
in thread Use more threads.

Note also the stack_size use line option added to threads as a result of this thread.

I quite routinely set this as low as 4096 without problems, and doing so I have had a proof of concept code with 3000 threads running in 4GB of memory.

Not that there are many good uses for using more than 3 or 4 times as many threads as you have cores. So unless you are running your code on a system with 32 cores or more, seeking to run so many threads is usually a sign of naive coding.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.


Comment on Re^2: Use more threads.
Download Code
Re^3: Use more threads.
by password (Sexton) on Apr 29, 2013 at 07:23 UTC

    I agree completely with what you said about using many threads (or maybe using any threads at all) but what I've written is a proxy checker (with GUI), so most of the time my threads are waiting for a response or a timeout.

    My next idea is to see if I can create 100 threads any faster, so I'll tighten it up as much as it is possible, and try that stack_size option. Although my coding is a bit naive and far from professional, my only fear is that I'm inventing the wheel sometimes. he-he.

      t I've written is a proxy checker (with GUI), so most of the time my threads are waiting for a response or a timeout.

      Starting new threads for each connection is naive. And wasteful.

      A pool of threads and a queue of requests is more efficient (of time and resources), and actually easier to program and reason about.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

        Of course, a pool of threads is what I have. I create it even before any jobs are enqueued, but still the process of creating threads eats up 100% of CPU, and creating 100 threads takes a long time (about 20-22 seconds). After threads are created, the CPU usage drops.

        Maybe I'm running out of limits of my system, after all it's a P4 3.2GHz 9 y.o. computer with 3GB of DDR type slowwww, but I would have no excuse 9 years ago, so there's a possibility I am the bottleneck! :p

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1031039]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (10)
As of 2014-07-25 07:00 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (169 votes), past polls