Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??

So I was working on multi-threading a little tool the last couple of days, but I didn't need any sharing or anything of the sort. I just needed to assign one thread per one file that needed to be parsed. The reason I am writing this is because the documentation can be a little misleading. The one consistent error I saw people make while googling is because of detach(). From the documentation, "Once a thread is detached, it'll run until it's finished; then Perl will clean up after it automatically." When I first started this little tool I thought this was just what I needed. However, what the documentation doesn't say is that if the Main Perl Process closes before a detached thread does, all detached threads are closed as well, regardless of completion or not. Looking back now, it makes sense, but the documentation at the time gave me the impression that Perl created threads not reaped by the Main Thread(when detached). Not only that, but I read somewhere that the potential exists that the resources of the detached threads might also not be freed. So my issue was that all my threads were being created properly but the Main Perl Process was closing everything before my Worker threads could do their job. So here is my solution.

use threads; $num_threads; $thread_limit = xxx; foreach $file_to_execute (@array_of_files) { $thread = new Thread \&do_stuff, param1, $param2; @threadlist = threads->list(threads::running); $num_threads = $#threadlist; while($num_threads >= $thread_limit) { sleep(30); @threadlist = threads->list(threads::running); $num_threads = $#list; } } while($num_threads != -1) { sleep(1); @threadslist = threads->list(threads::running); $num_threads = $#list; } sub do_stuff{ ... }

So as you can see, I create a thread in a foreach loop for each item to be executed so to speak. The first while loop also limits how many threads are running at a time, useful for someone using a weaker computer. I just ran it on my work computer that has two quad cores and 16 gigs of ram without the while loop and it didn't break a sweat, but it might be useful for some people. The second while loop essentially prevents the Main Perl Process from closing while the array of running threads is not empty. Once it is emptied, the program quits...and yes I should have used strict. I was being lazy. I just remembered another reason I wanted to write this up. I was having an issue where

 threads->list(threads::running)

would increment to the max number of threads, but never decremented as threads finished. But now I can't think of how I solved it, or if there was ever truly something wrong in the first place besides my own fingers.

Either way this was a fun little exercise. I went from processing 200+ files(22.5 GB total, with 200+ threads!) in 1 1/2 hours to 23 minutes. The only reason it is as high as 23 minutes is because I have three 4 GB files that I have to read from top to bottom.So near the end I have four threads, the Main process and the three files being read and split apart. Don't think I can do anything about that in terms of threading. I am going to try 80+ gigs and see if I can break my computer.


In reply to Perl Threads Boss/Worker Example by joemaniaci

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others exploiting the Monastery: (6)
    As of 2014-10-23 18:05 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      For retirement, I am banking on:










      Results (126 votes), past polls