Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??

I'm really struggling on how to get threads to work properly in my code. I've searched the web for days, but I still can't figure out a way to handle what I want. Maybe I'm just stupid. But, if anyone here can give me an idea no how to do this, I truly appreciate it.

I need to run threads to compare 2 versions of texts (thousands of them.) I also need to set a thread limit so that those thousands of threads won't eat away my CPU and memory.

However, there are a couple of things that can happens in the threads, but currently my code can't handle them.

1. Some threads may be terminated unexpectedly. How can I catch those threads to re-run them or handle them gracefully?

2. When some threads hang, how can I time out those threads?

The Thread::Queue code I have below is something I saw somewhere in the web:
use File::Slurp; use String::Diff; use threads; use threads::shared; use Thread::Queue; my $thread_limit = 15; my $worker = Thread::Queue->new(); # Worker Threads my $result = Thread::Queue->new(); # Result Threads my @pool = map { threads->create(sub { my $tid = threads->tid; my ($worker, $result) = @_; while( my $item = $worker->dequeue ) { // code that I have that might die unexpectedly, or // hang for a long time // Some Example code here: my $old_text = 'something I got from text version 1'; my $new_text = 'something I got from text version 2'; %String::Diff::DEFAULT_MARKS = (remove_open => '<span clas +s="old">', remove_close => '</span>', append_open => '<span class="ne +w">', append_close => '</span>'); my ($old_diff, $new_diff) = String::Diff::diff($old_text, +$new_text); my $diff = "<html>\n<head>\n<meta http-equiv=\"Content-Typ +e\" content=\"text/html; charset=utf-8\">\n"; $diff .= "<style type='text/css'>\nspan { font-weight: bol +d; }\n.new { color: blue; }\n.old { color: red; }\n</style>\n</head>\ +n<body>\n"; $diff .= "<h1>New Version</h1>\n<p>$new_diff</p>\n<h1>Old +Version</h1>\n<p>$old_diff</p>\n"; $diff .= "</body>\n</html>\n"; write_file("filename.html", {binmode => ':utf8'}, $diff); // More code ... $result->enqueue($tid); } $result->enqueue(undef); }, $worker, $result); } 1 .. $thread_limit; # Queue up all work items (Say, I have 10000 texts to compare) $worker->enqueue($_) for (0 .. 10000); # Tell all workers there are no work items $worker->enqueue((undef) x $thread_limit); # Process the results for ( 1 .. $thread_limit) { while (my $tid = $result->dequeue()) { // Can I handle terminated threads or // time out threads here? // If so, how? } } # Clean up the threads $_->join() for @pool;

So, when a worker thread die unexpectedly, where can I insert code to handle it?

And, when a worker thread hangs for too long, where in the code can I time it out?

Another somewhat related question, it seems that String::Diff (Text::Diff::FormattedHTML too) doesn't work well in threads. The program will die with "Bus Error: 10" when running String::Diff in threads (usually happens after hundreds of threads are executed.) If I use Text::Diff instead, I don't see "Bus Error: 10". But Text:Diff is too obscure for my targeted users, is there a way to run String::Diff safely in threads?

Thank you very much


In reply to [Thread::Queue] How to handle unexpected termination and time out in threads by oobobe

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others examining the Monastery: (12)
    As of 2014-07-30 09:24 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      My favorite superfluous repetitious redundant duplicative phrase is:









      Results (230 votes), past polls