Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

[Thread::Queue] How to handle unexpected termination and time out in threads

by oobobe (Novice)
on Apr 09, 2014 at 08:36 UTC ( #1081624=perlquestion: print w/ replies, xml ) Need Help??
oobobe has asked for the wisdom of the Perl Monks concerning the following question:

I'm really struggling on how to get threads to work properly in my code. I've searched the web for days, but I still can't figure out a way to handle what I want. Maybe I'm just stupid. But, if anyone here can give me an idea no how to do this, I truly appreciate it.

I need to run threads to compare 2 versions of texts (thousands of them.) I also need to set a thread limit so that those thousands of threads won't eat away my CPU and memory.

However, there are a couple of things that can happens in the threads, but currently my code can't handle them.

1. Some threads may be terminated unexpectedly. How can I catch those threads to re-run them or handle them gracefully?

2. When some threads hang, how can I time out those threads?

The Thread::Queue code I have below is something I saw somewhere in the web:
use File::Slurp; use String::Diff; use threads; use threads::shared; use Thread::Queue; my $thread_limit = 15; my $worker = Thread::Queue->new(); # Worker Threads my $result = Thread::Queue->new(); # Result Threads my @pool = map { threads->create(sub { my $tid = threads->tid; my ($worker, $result) = @_; while( my $item = $worker->dequeue ) { // code that I have that might die unexpectedly, or // hang for a long time // Some Example code here: my $old_text = 'something I got from text version 1'; my $new_text = 'something I got from text version 2'; %String::Diff::DEFAULT_MARKS = (remove_open => '<span clas +s="old">', remove_close => '</span>', append_open => '<span class="ne +w">', append_close => '</span>'); my ($old_diff, $new_diff) = String::Diff::diff($old_text, +$new_text); my $diff = "<html>\n<head>\n<meta http-equiv=\"Content-Typ +e\" content=\"text/html; charset=utf-8\">\n"; $diff .= "<style type='text/css'>\nspan { font-weight: bol +d; }\n.new { color: blue; }\n.old { color: red; }\n</style>\n</head>\ +n<body>\n"; $diff .= "<h1>New Version</h1>\n<p>$new_diff</p>\n<h1>Old +Version</h1>\n<p>$old_diff</p>\n"; $diff .= "</body>\n</html>\n"; write_file("filename.html", {binmode => ':utf8'}, $diff); // More code ... $result->enqueue($tid); } $result->enqueue(undef); }, $worker, $result); } 1 .. $thread_limit; # Queue up all work items (Say, I have 10000 texts to compare) $worker->enqueue($_) for (0 .. 10000); # Tell all workers there are no work items $worker->enqueue((undef) x $thread_limit); # Process the results for ( 1 .. $thread_limit) { while (my $tid = $result->dequeue()) { // Can I handle terminated threads or // time out threads here? // If so, how? } } # Clean up the threads $_->join() for @pool;

So, when a worker thread die unexpectedly, where can I insert code to handle it?

And, when a worker thread hangs for too long, where in the code can I time it out?

Another somewhat related question, it seems that String::Diff (Text::Diff::FormattedHTML too) doesn't work well in threads. The program will die with "Bus Error: 10" when running String::Diff in threads (usually happens after hundreds of threads are executed.) If I use Text::Diff instead, I don't see "Bus Error: 10". But Text:Diff is too obscure for my targeted users, is there a way to run String::Diff safely in threads?

Thank you very much

Comment on [Thread::Queue] How to handle unexpected termination and time out in threads
Download Code
Re: [Thread::Queue] How to handle unexpected termination and time out in threads
by BrowserUk (Pope) on Apr 09, 2014 at 14:05 UTC
    So, when a worker thread die unexpectedly, where can I insert code to handle it?

    How do you handle it when a non-threaded program might die unexpectedly?

    A "non-threaded program", is actually a single-threaded program, and what works in one thread -- the main and only thread -- works just the same in any other thread. So do the same thing.

    And, when a worker thread hangs for too long, where in the code can I time it out?

    You know what I'm going to say right?

    How do you handle this in a single-threaded program?

    Do the same thing.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: [Thread::Queue] How to handle unexpected termination and time out in threads
by sundialsvc4 (Abbot) on Apr 09, 2014 at 15:29 UTC

    First of all, and most importantly, I would challenge your notion of using “thousands of threads.”   It won’t help, and it will hurt, because the processing that you describe is I/O-bound.   Don’t make that disk-drive move its read/write head around any more than you have to, because that is what costs you milliseconds apiece.   Most of the time, in situations that have been multithreaded but are not good candidates for it, all-but-one of your threads will be waiting for that disk-drive.   Remember that multithreading divides the CPU resource, and does not multiply anything.

    Secondly ... consider using the xargs command with the -p numprocs option to conveniently spawn numprocs instances of a simple single-threaded program which takes a filename as its command-line parameter and processes just that one file, then dies.   No funky programming required, and you can easily tweak the number of processes.   (You will find the “sweet spot” to be quite small, and it just might be “1.”)   Build a file containing a list of all the filenames to be processed.   (The ls command can do that.)

    For multithreaded processing in-general, each thread or process should be the true master of its own affairs, capable of trapping runtime errors and of using a timer-signal to interrupt what it is doing without losing control of the situation.   (Ditto if it is just a single-thread.)   Generally it is not a good idea to put a pistol to the head of any process because you do not know exactly what it was doing when its head exploded.   The process or thread should survive, no matter what, and exit graciously and cleanly.   The parent’s one and only responsibility is to mind the children (aye, ’twas ever thus ...), who do all the work as well as all the communication with the outside world.

    Finally, search CPAN to find the many existing frameworks that are already out there to do this very common thing.   (xargs -p being the simplest example.)   Don’t re-invent something that has already been well done.

Re: [Thread::Queue] How to handle unexpected termination and time out in threads
by Preceptor (Chaplain) on Apr 09, 2014 at 16:24 UTC

    First off - I'm going to offer the suggestion that an anonymous sub is a bad plan if it's more than a couple of lines long.

    There's one simple reason - because if you write a 'standard' sub, you can test your program non-threaded, and make all the 'concurrency issues' go away entirely.

    Thread::Queue supports 'end()'. This is a better way of handling the problem then enqueing a load of 'undefs'.

    When a thread is 'ended' then any (blocking) dequeue call returns 'undef' immediately. And a while loop construct like this:

    while ( my $item = $my_q -> dequeue() )

    will be 'undef' so your while loop will exit.

    Handling a loop that's prone to crashing out is typically accomplished by 'eval'. Use 'eval' to run the code, and then trap if it errors.

    Handling 'hanging' conditions is a bit harder - especially within a thread. I would normally suggest using an 'alarm' call and using $SIG{'ALRM'} to trap it. However, you need to be slightly careful in doing so, simply because thread -> join() will block this signal.

    A better approach would be to correct the error whereby your code hangs. It may be that you're just waiting forever on a dequeue() call or something similar. These go away if you close a thread to terminate it.

    I would also note - you 'end' your workers via queuing undef, and then go single threaded to parse your results. Why not try and sync your threads _before_ processing them, because then you can just 'end()' your results queue. Alternatively - have a separate 'result collator' thread (one only) and stuff each of your 'worker' threads into an array of thread references, which you can then 'join()' - meaning you always know when your workers have terminated (one way or another) before you close off your results processing.

    for ( 1..$num_workers ) { my $thr = threads -> create ( \&worker_thread ); push ( @worker_threads, $thr ); } foreach my $thr ( @worker_threads ) { $thr -> join(); }

    However as already observed - your process looks to be IO oriented, in that the most expensive thing you're doing is _probably_ reading/writing the files. More threads will not help particularly, and may be actively counterproductive - the more random IO pattern you use, the less efficient your storage subsystem will be able to cache, prefect and avoid disk seek contention.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1081624]
Approved by Corion
Front-paged by GotToBTru
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (9)
As of 2014-10-01 16:08 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    What is your favourite meta-syntactic variable name?














    Results (29 votes), past polls