|
Re: Threading with a twist
by ikegami (Patriarch) on Oct 03, 2017 at 07:46 UTC
|
As previously mentioned, process STDIN and $rx_q in separate threads. But also, use dequeue instead of dequeue_nb.
use strict;
use warnings qw( all );
use threads;
use Thread::Queue qw( ); # 3.01+
use constant NUM_WORKERS => 10;
sub worker {
my ($job) = @_;
return "[$job]";
}
sub collector {
my ($job) = @_;
print "{$job}\n";
}
{
my $tx_q = Thread::Queue->new();
my $rx_q = Thread::Queue->new();
my @workers;
for (1..NUM_WORKERS) {
push @workers, async {
while (defined( my $job = $tx_q->dequeue() )) {
$rx_q->enqueue(worker($job));
}
};
}
my $collector_thread = async {
while (defined( my $job = $rx_q->dequeue() )) {
collector($job);
}
};
while (my $job = <>) {
chomp($job);
$tx_q->enqueue($job);
}
$tx_q->end();
$_->join() for @workers;
$rx_q->end();
$collector_thread->join();
}
| [reply]
[d/l]
[select] |
Re: Threading with a twist -- a panorama
by Discipulus (Canon) on Oct 03, 2017 at 07:23 UTC
|
Hello scorpion7 and welcome to the monastery and to the wonderful world of Perl!
you are asking for some guindance and I must admit I'm not able to provide it, but in the monastery you can find very good suggestions about this matter. Even if I'm not able to provide such guidance I can suggest you something because I'm following parallel perl programming since years, even if i produced almost nothing in this field.
First of all: is possible to produce full working perl parallel applications but the path is full of possible pitfalls. At some point perl threading was oficially discouraged because was too easy make it the wrong way (*).
But is definetively possible to produce good working code but you must take inspiration (read: copy) from the rigth authors. Here at perlmonks we have definetively at least two incredibly neat parallel perl programmers: BrowserUk and zentara.
While the first is more used to work on windows systems the latter almost exclusively works on Linux. Super Search their posts about parallel programming and you'll get a goldmine of neat and smart solutions and many insigths about possible pitfalls.
This does not means that no other monks here is a parallel programming expert, but the two above are eminent for offering full working examples (for example karlgoethebier's serie Threads From Hell is also very interesting).
Many links are present in my homenode about this matter, not only by the two monks above but also by others experimenting about parallel programming.
Then there is the framework: read my meditation parallel programming and MCE - hubris and the frame about it.
Infact a recently appeared monk, marioroy has produced a wonderful module suit to help building parallel applications with the full support for windows systems and many other goodies: MCE Many-Core Engine for Perl.
marioroy is very active and responsive here at perlmonks and you can read with profit all his posts here because all are about parallel programming and he offers many times complete examples and benchmarks between different approaches.
I hope the above is interesting for you and I hope you'll share your findings and progress.
L*
Note (*) I think my opinion about the possibility to do parallel programming in Perl seems to me clearly exposed in the present thread and in the linked one: however it also seems that to quote a, even if discutible, part of official documentation attracts critics as below expressed.
I see no problem in what I said if examinated in the context. Anyone can think what they want.
L*
There are no rules, there are no thumbs..
Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
| [reply]
[d/l] |
|
|
some point perl threading was oficially discouraged because was too easy make it the wrong way.
Bah, if you do it wrong --not reuse threads-- it's just going to be a bit slower. Yup, that's all there is to that warning. Just use a worker model and you're fine.
Someone threw a tantrum when they wrong that warning, and it's poisoning the community.
| [reply] |
|
|
| [reply] |
|
|
|
|
|
|
|
|
A reply falls below the community's threshold of quality. You may see it by logging in.
|
|
|
| [reply] |
Re: Threading with a twist
by Random_Walk (Prior) on Oct 03, 2017 at 05:17 UTC
|
Boss:
- Waits for <STDIN>
- Appends manipulated input to $TXQUEUE->enqueue()
- Monitor $RXQUEUE and <STDIN> - simultaneously???
- Sends <STDOUT> while(my $RESULT=$RXQUEUE->dequeue_nb()), until $RXQUEUE is empty. $RESULT is sent "AS-IS"
I would split that boss into two threads. A dispatcher listening on <STDIN> handing of work to workers and a publisher de-queueing the results from workers and publishing them to <STDOUT>
Cheers,
R.
Pereant, qui ante nos nostra dixerunt!
| [reply]
[d/l]
[select] |
|
|
| [reply] |
|
|
Because KISS
The design in the OP gives two very disparate jobs to the 'Boss'. It must flip between checking if there are results on the results queue and handling them, then seeing if there is anything to read on <STDIN> and passing it off to workers. The OP is talking about using non blocking calls and sleeps to do this. The complexity clearly recognised by the multiplicity of punctuation marks ... Monitor $RXQUEUE and <STDIN> - simultaneously???
If we split these two functions then we have two simple tasks.
- Listen on <STDIN> and put work on worker queue.
- Listen on results queue and publish results
both tasks can be written with blocking calls ensuring they sleep efficiently until they have work, then spring to work with alacrity when there is something to do.
Cheers,
R.
Pereant, qui ante nos nostra dixerunt!
| [reply]
[d/l]
[select] |
|
|
|
|
Re: Threading with a twist
by Anonymous Monk on Oct 03, 2017 at 04:50 UTC
|
Hi
Have seen a lot of posts about not using Perl library 'X' for pools/etc. For 5.24 (or better) is that still the prevailing perspective? (eg: no better mouse trap)
:) what are you asking?
I've seen all the posts, you don't know what you're asking, and that is an answer in itself
Have seen the use of "usleep", is that the best method for "idling" a worker thread?
That is pretty much the only method for idiling, nothing else is idle.
Is there a way to achieve the benefit of the while loop (avoid CPU cycle consumption while idle), but then issue of providing results to <STDOUT> the moment (or very close to when) they are available? What is a moment? Thats not just a question, that is the answer , that is how long you should (ask to) idle.
| [reply] |
Re: Threading with a twist
by Anonymous Monk on Oct 03, 2017 at 15:30 UTC
|
Also bear in mind that workflow-management tools already exist in the CPAN library! You are re-inventing a very old wheel. | [reply] |
|
|
While it might be an "old wheel", the value in what is learned - is believed to be significant and useful to advance one's ability with Perl.
| [reply] |
