|
petesmiley has asked for the wisdom of the Perl Monks concerning the following question:
Hey all,
I have a piece of code here that uses Thread::Queue. I wrote it as a test case when I was rooting around with Devel::Leak. This sucker will eat every ounce of memory I have and then my lunch if I let it. Can anyone tell me why?
For some reason, it seems like dequeue does not free up the memory when it dequeues. Should I be using a different module?
smiles
#!/usr/bin/perl
use strict;
use threads;
use Thread::Queue;
my $q = Thread::Queue->new();
for (1..3) {
$q->enqueue('A' x 100);
threads->new(\&ttest)
}
my @threadlist = threads->list;
sleep 100;
$q->enqueue(undef) for @threadlist;
$_->join for @threadlist;
sub ttest {
while (my $item = $q->dequeue) {
my $cnt = $q->pending;
print "$cnt\n";
$q->enqueue('A' x 100);
}
}
|
Re: Thread::Queue and Memory
by gmpassos (Priest) on Feb 21, 2003 at 02:21 UTC
|
Fisrt, take a look at POD for thread on Perl:
Perl threads tutorial (threads)
Old Perl threads tutorial (Thread)
Note that Perl 5.7+ has a new thread implementation, called ithreads, in the module "threads", and you still can access the old implementation, in the module Thread.
I don't know if the module Thread::Queue will work with ithreads (threads), since this module was inside the old way (Thread)!
You can see in the end of Thread POD, this comments on "See Also":
threads::shared (not available with 5005threads)
attributes, Thread::Queue, Thread::Semaphore, Thread::Specific (not available with ithreads)
Take a look at threads::share and the lock() function for a safe way to work with a shared list (since Thread::Queue is for a shared data structure, like a list).
Graciliano M. P.
"The creativity is the expression of the liberty". | [reply] |
|
|
Doh! Slaps head. Of course.
My confusion was somewhat caused by the examples listed in the tutorials. Not only that, but the Thread::Queue POD aludes to using "threads" not "Threads".
Thanx Chief
Update: Now that I've had time to think about it, it's not just locking I need, its cond_wait() and cond_signal(). Which is why I resorted to Thread::Queue fo simplification in the first place.
If you look at the POD for Thread::Queue
it refers to the ithreads implimentation. Not the old perl threads implimentation.
- So, I either don't understand Thread::Queue properly and it is supposed to work with ithreads, and the docs on the old Threads POD needs to be updated.
- Or, Thread::Queue is not supposed to work with ithreads and there is something fuzzy with the docs and there needs to be a clarification.
- Or, there is something actually wrong with Thread::Queue, it is supposed to work with ithreads, and the docs for the old Threads implimentation needs to be fixed.
Somehow, I think that last little line in the old Threads POD is not up to date according to the current documentation for perl 5.8.
| [reply] |
Re: Thread::Queue and Memory
by JamesNC (Chaplain) on Feb 20, 2003 at 21:46 UTC
|
I think it is because the queue gets eaten too quickly..look at this:
#!/perl/bin/perl
use strict;
use threads;
use Thread::Queue;
my @threads;
my $thread;
my $q = Thread::Queue->new();
$q->enqueue(1..100);
print "Items in the queue: ",$q->pending,"\n";
for (1..5) {
push @threads, threads->new(\&ttest);
print "spawned thread:";
}
foreach $thread (@threads){
$thread->join;
}
sub ttest {
while (my $cnt = $q->pending) {
my $item = $q->dequeue;
print "$cnt\n";
last if $cnt == 0 ;
}
}
cheers :0)
JamesNC | [reply]
[d/l] |
|
|
I'm not sure I understand what you are trying to show. I currently have a much larger sample of what I wrote. And it quite happily chews up memory with a full queue (more than 0). The queue always shows the correct size but for some reason it keeps eating more memory.
The queue you have here would show the same problem I have when I analyze it under Devel::Leak I still see it allocating extra scalars every time it dequeues.
Point being, my little piece of code does what my poor brain says will be just of few megs of ram. But instead it takes 80+.
| [reply] |
|
|
Ok, you were asking about the known memory leak in threads then?
Perl Threads Leak
If you think you know what's causing it, the threads author is mentioned in the link I believe. I don't know how many people are working on this, if you think you know enough to debug it, I am sure they would be very happy to let you have a go at it.
| [reply] |
Re: Thread::Queue and Memory
by clairudjinn (Beadle) on Feb 20, 2003 at 22:54 UTC
|
As a point of unrelated discussion, I've been looking at closures lately in an effort to understand what they are all about, so I'm wondering if I have correctly identified ttest() as a closure. | [reply] |
|
|
| [reply] |
Re: Thread::Queue and Memory
by petesmiley (Friar) on Feb 21, 2003 at 17:22 UTC
|
| [reply] |
|
|
