I'm really struggling on how to get threads to work properly in my code. I've searched the web for days, but I still can't figure out a way to handle what I want. Maybe I'm just stupid. But, if anyone here can give me an idea no how to do this, I truly appreciate it.
I need to run threads to compare 2 versions of texts (thousands of them.) I also need to set a thread limit so that those thousands of threads won't eat away my CPU and memory.
However, there are a couple of things that can happens in the threads, but currently my code can't handle them.
1. Some threads may be terminated unexpectedly. How can I catch those threads to re-run them or handle them gracefully?
2. When some threads hang, how can I time out those threads?
The Thread::Queue code I have below is something I saw somewhere in the web:
use File::Slurp;
use String::Diff;
use threads;
use threads::shared;
use Thread::Queue;
my $thread_limit = 15;
my $worker = Thread::Queue->new(); # Worker Threads
my $result = Thread::Queue->new(); # Result Threads
my @pool = map {
threads->create(sub {
my $tid = threads->tid;
my ($worker, $result) = @_;
while( my $item = $worker->dequeue ) {
// code that I have that might die unexpectedly, or
// hang for a long time
// Some Example code here:
my $old_text = 'something I got from text version 1';
my $new_text = 'something I got from text version 2';
%String::Diff::DEFAULT_MARKS = (remove_open => '<span clas
+s="old">', remove_close => '</span>', append_open => '<span class="ne
+w">', append_close => '</span>');
my ($old_diff, $new_diff) = String::Diff::diff($old_text,
+$new_text);
my $diff = "<html>\n<head>\n<meta http-equiv=\"Content-Typ
+e\" content=\"text/html; charset=utf-8\">\n";
$diff .= "<style type='text/css'>\nspan { font-weight: bol
+d; }\n.new { color: blue; }\n.old { color: red; }\n</style>\n</head>\
+n<body>\n";
$diff .= "<h1>New Version</h1>\n<p>$new_diff</p>\n<h1>Old
+Version</h1>\n<p>$old_diff</p>\n";
$diff .= "</body>\n</html>\n";
write_file("filename.html", {binmode => ':utf8'}, $diff);
// More code ...
$result->enqueue($tid);
}
$result->enqueue(undef);
}, $worker, $result);
} 1 .. $thread_limit;
# Queue up all work items (Say, I have 10000 texts to compare)
$worker->enqueue($_) for (0 .. 10000);
# Tell all workers there are no work items
$worker->enqueue((undef) x $thread_limit);
# Process the results
for ( 1 .. $thread_limit) {
while (my $tid = $result->dequeue()) {
// Can I handle terminated threads or
// time out threads here?
// If so, how?
}
}
# Clean up the threads
$_->join() for @pool;
So, when a worker thread die unexpectedly, where can I insert code to handle it?
And, when a worker thread hangs for too long, where in the code can I time it out?
Another somewhat related question, it seems that String::Diff (Text::Diff::FormattedHTML too) doesn't work well in threads. The program will die with "Bus Error: 10" when running String::Diff in threads (usually happens after hundreds of threads are executed.) If I use Text::Diff instead, I don't see "Bus Error: 10". But Text:Diff is too obscure for my targeted users, is there a way to run String::Diff safely in threads?
Thank you very much
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.