Well, what I need to do is a bit more complicated than that ;-).
Basically, I have a very complex object of class WeBiText. The object carries out a complex search algorithm, part of which, called the L1 Search, does a number of searches on Yahoo, that produces hits to be processed by the rest of the search algorithm. Because the Yahoo searches take a long time to execute, I want the L1 search to be done in a separate thread, and in such a way that it communicates the hits it finds back to the main search algorithm as soon as it finds them. That way, the main search doesn't have to wait for the complete set of L1 hits before it starts doing its work.
There are issues that complicates things.
- WeBiText is not thread safe, because it needs some modules which are not thread-safe.
- A consequence of this is that all threads in the thread pool must be created before any instance of WeBiText is created.
- Another consequence is that WeBiText instances can't be shared between threads and must instead be serialized and deserialized across threads.
- Also, I run hundreds of automated tests on WeBiText, using PerlUnit
A consequence of points 2 and 4, is that threads must be "reusable", otherwise I would have to create hundreds of threads at the start, before I run the tests which create WeBiText instances. So the pool must allow a client to "reserve" a thread and later on release it so it can be reused. This is one functionality that Thread::Pool provides.
Also, a consequence of point 3 is that the L1 search cannot directly notify the WeBiText instance running in the main thread when it finds a new L1 hit, because it doesn't receive a shared instance of that WeBiText instances. The best that can be done is for the L1 thread to receive a serialization of the master thread's WeBiText instance, and use that serialization to recreate the instance in the L1 search thread. But any change to the state of that recreated instance won't affect the state of the original instance running in the main thread.
My solution to this is for the L1 search to put its hits on an "outgoing" queue that is shared between the two threads. The problem is that both threads must know about this queue. But since WeBiText instances must be created after the threads are created, it means that any queue that it creates cannot be shared with those threads. So, it's the pools responsibility to provide a results queue that is associated with a particular thread. Those hits queues must be reserved and released just like the threads.
Note that this behaviour is not provided by Thread::Pool, and I need to implement it myself anyway. But since Thread::Pool already provides reusable threads, I thought I'd start from it. Maybe that was a mistake.
As you can see, this is not something that can be implemented in a couple of lines ;-).
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
