Tk and Threads (again)

[Ace128]

Well, hello there!

Let's see now... I have this main script starting up threads and then Tk (yes, done alot of research on the matter. Need to start threads before Tk). For each thread there is a Thread::Queue::Any. I have a plugin idiom.. Well, code for starting up threads:

foreach (keys %plugins) {
    my $thread_queue = Thread::Queue::Any->new;
    my @thread_subs;
    $plugins{$_}->SUBS_AS_THREADS(\@thread_subs, $thread_queue);
    foreach my $sub (@thread_subs) {
        async { $sub->($plugins{$_}, \%plugins, $thread_queue, $mw) };
    }
}
...
...
MainLoop;

@thread_subs is an array of subrefs containing all subs(refs) a surtain plugin wanna start up as a new thread.
$plugins{$_} is the current plugin (I loop through them all here.) Below is an example of a threadsub:

sub _thread_sub_in_a_plugin {

    my $self         = shift;
    my $plugins_ref  = shift;
    my $thread_queue = shift;
    my $mw           = shift;

    #$self->{'thread_queue'} = $thread_queue;
    $$plugins_ref{'imagePlugin'}->{'thread_queue'} = $thread_queue;
    # The 2 above same thing, really...

    while (1) {

        my ($full_path) = $thread_queue->dequeue;


    ...
    }
}

As you can see, I found this solution quite nice, since I use Tk, and wanna use threads. Since I dunno how many threads a plugin wanna start (and more importantly, IF a plugin wanna start a thread), I have a sub that sets refrences into an array to subs a surtain thread wanna have as threads (@thread_subs).

Now, notice the $mw above. The idea was that if a plugin wanted to do some nice GUI stuff (since this is the main point having it as a seperate thread, so it all goes smoothly), the $mw was passed in so it could do something like $mw->toplevel, and use that to do stuff on. However, everything that is created within this thread and has to do with Tk, makes me get the error "Free to wrong pool 2741b8 not 64a0778 at ..." thing, when it wanna destruct something created in there (why? I mean, nothing should get destructed, since the thread isnt done (while (1) ) ).
The main reason I do this is, since its an image plugin, I wanted it to generate an image based on what the path is it gets from the $thread_queue.
Now, I had 3 ideas for solutions:

1. If it was possible, let the other thread have it's own Tk Mainloop and everything. No collision with the main one. This seems impossible to do.

2a. Return the data created (by imagequick in this case), and put it in the Toplevel in the main thread. This would mean that the imagecreation process would be in a seperate thread, and then pass back the data through Queue::Any, and put into the Toplevel object. However, this still means that the main thread has to wait until the imagequick thread is done, so it really seems pointless this too, but I would avoid the destruction problem...

2b. I was also thinking that I could, however, have a $mw->repeat(10, ...) set, to update if there was something in the $thread_queue... However, I dont really like that idea. Such a waste of CPU and not "realtime"... I would prefer that something happends when thread is done! Or what do you guys think about it repeating every 10 ms? Checking if there is something new in the $thread_queue.

3. I was also looking into letting the imagethread create a GUI using wxPerl instead! This could work (although I havent manage to make it work yet...) But, I would prefer not to use it, since I already have Tk... :) Anyone here managed to use wx and tk at the same time? (In two seperate threads then I suppose, since both have a ->Mainloop call, that waits for the GUI stuff to end...)

Hope I didnt miss anything. Im so into this, its all very clear to me what the problem is. Explaining it to others, avoiding telling a shortstory, is tricky ;)
The idea:

Main(Tk)   imagePlugin
  |
  |
  |------------*
  |            |
  |     generate and show image
  |
  |

Ideas? Comments? Suggestions?

Thanks!
Ace

[zentara]

You are running into the same roadblock everyone who runs threads with Tk encounters. You cannot pass a Tk widget to a thread, even if it isn't going to do anything. So that accounts for your "wrong pool" error, and there isn't anyway around it with Tk.

Now the question is "how to do it, while keeping the GUI thread-safe?" The #1 idea could be done by forking off and communicating with sockets....but that is probably harder than the threads.

#2b is the way I usually do it. Have shared variables, which can be set by the threads and read by the Main-GUI thread with a timer. When the GUI sees the flag, it processes the data for display, and possibly closes off the thread( or puts it to sleep for later use). A timer is not that much over head, and is a clean easy-to-use solution.

You can mix Gtk2( what Wx is based on) and Tk. But you will still have to use a timer to call the "do-one-loop" iteration of the secondary gui; that keeps the secondary gui from freezing. Here is a Tk MainLoop running a Gtk2 loop. (It can be done the other way too)

#!/usr/bin/perl -w
use strict;
use Gtk2;
use Tk;

my $mw = MainWindow->new(-title=>'Tk Window');

Gtk2->init;

my $window = Gtk2::Window->new('toplevel');
$window->set_title('Gtk2 Window');
my $glabel = Gtk2::Label->new("This is a  Gtk2 Label");
$window->add($glabel);
$window->show_all;

my $tktimer = $mw->repeat(10, sub{      
   Gtk2->main_iteration while Gtk2->events_pending;
  });


$mw->Button(-text=>'     Quit     ',
       -command => sub{exit}
        )->pack();

MainLoop;
########################################
__END__

You might want to look at the newest Gtk2-Perl threads demo. They have an example that might be what you are looking for. They have a "thread-safe" mode that lets you share some widgets in threads, and "enter" and "leave" the thread to update the threaded widgets.

[renodino]

I suspect you'll need to use a form of apartment threading (I call it zone threading) where multiple objects run in their own thread (in this case, Tk::MainWindow and all its descendents).

To expose the $mw object to the child threads, you'll need to create a proxy object (possibly as a threads::shared object) to wrap the Thread::Queue::Any back to the Tk "zone". You'll probably need to use AUTOLOAD within the proxy, as enumerating the entire set of calls possible on $mw is pretty difficult.

I'll offer Thread::Apartment::Client as an implementation of the concept.

However, Tk presents an add'l sticky issue: its a heavy user of closures, and passing closures between threads is not possible (and the closure should probably always run in its originating thread anyway). I hope to provide a solution to this in Thread::Apartment via proxied closures.

[Ace128]

Is this for having a second MainLoop? (another seperate Tk thingy) If so, nice! Since the only thing they share is the one scalar from $thread_queue (which is just the path to where the file is). Rest is created in the thread and shown. No need to return anything if the thread can do it all!

[renodino]

Is this for having a second MainLoop?

No, its just a means to simplify the interaction between threads; you shouldn't need a 2nd MainLoop, just a $mw->repeat() call within the Tk thread that goes off to test the Thread::Queue every 100 msecs or so. The rest of the threads use the client proxy as a surrogate for making calls directly on $mw. (Of course, there's the little matter of marshalling params to the Tk thread, and getting results back...again, see Thread::Apartment for its marshal/unmarshal methods.)

Hopefully, I'll find time in the next couple of months to finalize Thread::Apartment support for closures, plus a few other things, and then implement a Tk::Threaded that will make all of this much simpler.

[Errto]

Just as a follow-on to zentara's excellent suggestion, I would mention that $mw->repeat(10, ...) shouldn't, I feel, be considered a bad idea. I mean, yes it's "wasteful" in the sense that there is an overhead imposed for the large majority of the time when it doesn't find anything in the queue, but at the same time, threads impose a rather substantial overhead penalty as well because they force Perl to implement preemptive multitasking within a single process, which is no mean feat.

I don't have hard numbers, but I can support this anecdotally by observing that I tried implementing something sort of like this where I needed an LWP GET request to update a Tk progress bar. I got a much smoother result by having the :content_cb subroutine from my LWP::UserAgent call $mw->update directly than by having LWP and the Tk MainLoop running in separate threads.

[BrowserUk]

but at the same time, threads impose a rather substantial overhead penalty as well because they force Perl to implement preemptive multitasking within a single process, which is no mean feat.

Not so. All the preemption and multitasking is done by entirely by the operating system, regardless of which OS it is running on. Perl uses either POSIX threads or Win32 native threads.

Unlike Java and several other langauges which do implement their own mini-schedulers within each process, Perl leaves all the scheduling to the OS.

[Errto]

Huh. Didn't realize that. I guess after all of the (mostly negative) material I've read here about Perl threads I had assumed that they were totally in-process. Then it's possible I was doing something wrong to produce the clunky thread behavior I was seeing (with the timeslices not being distributed very well), or else Windows threads just do that. BTW, Java can be built with what they call "green" threads which are totally in-process but these days the default build uses native threads including NPTL on Linux.

[zentara]

As an after-thought, you could fork separate scripts and communicate through shared memory. It is actually very fast to share this way. Check out Tk-shared-mem-ipc for an example of 3 forked Tk scripts, interacting with one another through shared memory segments.

[Ace128]

Nice! However, I always strive for multios support. And, I couldn't find IPC::SysV for Windows!

[zentara]

I don't use Windows, but have seen in other posts, that win32 has a module called memmap, and Tie::Win32MemMap which essentially does the same thing as IPC::SysV. So you could have different routines for each OS, but threads is probably simpler.