Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Re^4: Use more threads.

by BrowserUk (Pope)
on Apr 29, 2013 at 08:36 UTC ( #1031185=note: print w/ replies, xml ) Need Help??


in reply to Re^3: Use more threads.
in thread Use more threads.

t I've written is a proxy checker (with GUI), so most of the time my threads are waiting for a response or a timeout.

Starting new threads for each connection is naive. And wasteful.

A pool of threads and a queue of requests is more efficient (of time and resources), and actually easier to program and reason about.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.


Comment on Re^4: Use more threads.
Re^5: Use more threads.
by password (Sexton) on Apr 30, 2013 at 00:26 UTC

    Of course, a pool of threads is what I have. I create it even before any jobs are enqueued, but still the process of creating threads eats up 100% of CPU, and creating 100 threads takes a long time (about 20-22 seconds). After threads are created, the CPU usage drops.

    Maybe I'm running out of limits of my system, after all it's a P4 3.2GHz 9 y.o. computer with 3GB of DDR type slowwww, but I would have no excuse 9 years ago, so there's a possibility I am the bottleneck! :p

      This starts 100 threads in < 1/2 a second:

      #! perl -slw use strict; use Time::HiRes qw[ time ]; use threads stack_size => 4096; my $start = time; async { #printf "%u starting\n", threads->tid; sleep 1e3 } ->detach for 1 .. 100; printf "Took %f seconds\n", time() - $start; __END__ C:\test>junk Took 0.445518 seconds

      Those are do nothing threads, but still, you must be doing something weird. Post your code.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

        I just used your code, it took 1.37 seconds. Then I added 9 lines of "use Modules" after "use strict" and now it takes 12.3 seconds:

        use Socket; use IO::Socket; use DBI; use DBD::mysql; use Tk; use Tk::JPEG; use Tk::Table; use Tk::ProgressBarPlus; use LWP::UserAgent;

        Adding "use LWP::UserAgent;" alone makes it run for over 5 seconds.

        The complete code:

        #! perl -slw use strict; use Socket; use IO::Socket; use DBI; use DBD::mysql; use Tk; use Tk::JPEG; use Tk::Table; use Tk::ProgressBarPlus; use LWP::UserAgent; use Time::HiRes qw[ time ]; use threads stack_size => 4096; my $start = time; async { #printf "%u starting\n", threads->tid; sleep 1e3 } ->detach for 1 .. 100; printf "Took %f seconds\n", time() - $start;

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1031185]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (5)
As of 2014-09-22 22:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (206 votes), past polls