Passing objects between threads...any solutions ?

[kimanaw]

Seeking some threads wisdom, or at least a special dispensation...

As the doco's say, thread shared objects can't be blessed. And only scalars can be passed on Thread::Queues.

I've got a multithreaded server architecture (sortof a publish/subscribe, or perhaps observer/observable pattern). A request comes in, I turn it into a perl object, then have to give multiple configurable modules a chance to fiddle with it, possibly with long latency operations. When they're all done fiddling, the resulting request object gets marshalled and forwarded out the backend.

Problem is, I need a way to post the request object to the fiddler modules so they can each play their tune, and the request object just waits til everyone finishes.

Since I can't share the blessed object with the fiddler threads, the only thing I can think of is to somehow marshall the interesting bits out of the request object into a shared hash, and have each fiddler thread create a "shadow" request object to provide fiddling methods, then send it all back up to the original request object again. (Slow, painful, very likely error prone).

Can anyone recommend an alternative method/pattern to accomplish this (aside from jumping to Javayukfoo) ?

[BrowserUk]

How brave are you?

The code below uses a technique I've been playing with to share two instances of an object between 10 threads that each modify whichever instance they get 100 times before terminating. The main thread then displays the values which will be whatever values were last set into them by whichever thread accessed them last. The program runs reliably under 5.8.4 on my (single cpu) system with no idication of errors, traps or memory leakage.

Read more... (2 kB)

The basis of idea is that

1. re-bless the object instance into each thread which avoid the cross-thread method calling problem.
2. You store references to shared instance variables within the hash object and dereference them to read or modify them.
3. You embed a reference to a shared object that is used as a semaphore (via lock) into each instance and lock this in each method before accessing the instance data.

I haven't had the facilities to test this on a multi-processor system, but I think it should be safe. Any feedback you can let me have would be much appreciated.

[kimanaw]

OK, lemme see if I follow the plot...
1. You create a pool of "resource" objects whose members are all shared

2. you pass the objects to the threads when they are created

3. on creation, the thread dup's the resource object, thereby inheriting the shared members (albeit inside a new version of the object)

My only wrinkle is that I'm using a thread pool...but if I use a resource pool, I can instantiate the resource pool into each thread in the thread pool at startup, so I just need to add some scalar ID to each resource thread that can be used to direct a thread to whichever resource object to fiddle with.

Hmmm, intriguing...

I will definitely give it a shot. It may be a bit of a memory pig, but actually may also help limit the chances for memory leaks (kinda like building all Perl/Tk widgets at startup, and withdrawing til needed...kills memory leak bugs DEAD!)

PS. I do in fact have a dual CPU system (running Win2K); don't know when I'll get a chance to test this on it, (lots more code to implement/test before that), but I definitely owe you a barley pop...

And here's hoping Ponie solves this issue...

Many thanks...

[BrowserUk]

but I definitely owe you a barley pop...

If you could post (on your scratchpad) the results from running my test code on your dual processor and /msg me I'd be very happy.

1. You create a pool of "resource" objects whose members are all shared

2. ...

3. ...

That's not quite right, but I'm a too tired to correct it now. Check back later/tomorrow (before you start coding based on this) and I'll try to clarify.

[ikegami]

Since you can pass strings around, you could serialize the object into a string and deserialize it the next thread (assuming it doesn't contain handles or other special stuff). There's a module to do that, but its name escapes me.

[thcsoft]

Storable.pm performs the serialization.

[johnnywang]

I'd like to suggest you take a look at POE, a single threaded event-driven framework, (almost)perfect for your kind of problem. The reason I say "almost" is that I haven't been able to figure out how to do publish/subscribe with it, its default is to raise an event, and register one handle. I'm sure it's doable. But it certainly solves your thread-sharing problem.

[Anonymous Monk]

Why pass an object around? DISTRIBUTE it over the fiddler-tasks! RFC677 will tell you how. Even if it talks about a database it really is what a perl programmer would call a hash.

[kimanaw]

Er, don't see how it applies (at least to my situation). Keep in mind that each fiddler may need to see what other fiddlers are doing (or perhaps have done), so there's potential pipelining required...and, ultimately, my solution will in fact lead to distributing the resource to all the fiddlers.