Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Isolating dynamically loaded modules with Thread::Isolate.

by gmpassos (Priest)
on Jan 29, 2005 at 21:48 UTC ( [id://426297]=perlmeditation: print w/replies, xml ) Need Help??

Felow monks,

Recently I was working a lot with threads, producing a group of services that should work 24h per day, 7 days per week. This services are multithread since they access the internet doing some tasks, so they need to split the tasks in many threads to do the job faster and use better the hardware and link of our servers. But we have the problem that each thread in Perl uses more and more memory, so I want to share my experience on trying to fix that.

To do this tasks over the internet we use WWW::Mechanize that make our life much more easy, specially when we need to login in some site.

Today Perl 5.8.6 is a real solution for multithread services, but we have a big problem, memory usage. Our servers don't run only one service, but many services, and each thread of WWW::Mechanize uses a lot of memory! So, we can't start much more than 5 threads per process, since with 5 threads the service uses 60Mb+-!

Looking for a solution I have created Thread::Isolate, where the main idea is to create a thread than can be called externally, so, I can load modules from outside and call them. With that I can organize my programs in some threads, each of them with different resources and only duplicate threads/resources that will need to run in parallel.

Here's a simple sample of Thread::Isolate:

use Thread::Isolate ; my $thi = Thread::Isolate->new() ; $thi->use('Data::Dumper') ; print $thi->call('Data::Dumper::Dumper' , [123 , 456]) ;
Also I can do detached (unsynchronous) calls:
use Thread::Isolate ; $|=1 ; my $thi = Thread::Isolate->new() ; $thi->use('LWP::Simple'); my $job = $thi->call_detached('LWP::Simple::get' , 'http://www.perlm +onks.com/') ; while( $job->is_running ) { print "." ; sleep(1); } $html = $job->returned ; die( $thi->err ) if $thi->err ; print $html ;
Looking the code above we can see that we are not only isolating LWP::Simple, but we also can run a code until get() returns something.

Thread::Isolate was inspirated on Thread::Tie::Thread by Elizabeth Mattijsen (liz), that did something like that to make Thread::Tie. Than I saw that I could use that to organize my multithread programs, so I created a new module with more resources and an intuitive interface, enjoy! ;-P

Like any new module it still needs tests, and a module that works with multithread need more tests, so, I'm asking for you monks to test it and report your experience with it here. Thanks in advance.

CPAN should put it soon at:
http://www.linorg.usp.br/CPAN/authors/id/G/GM/GMPASSOS/

Graciliano M. P.
"Creativity is the expression of liberty".

Replies are listed 'Best First'.
Re: Isolating dynamically loaded modules with Thread::Isolate.
by holli (Abbot) on Jan 29, 2005 at 22:15 UTC
Re: Isolating dynamically loaded modules with Thread::Isolate.
by bart (Canon) on Jan 29, 2005 at 22:19 UTC
    I don't understand why you have two different variables, $thi and $job, for the same isolated thread. Does this imply you can run two different jobs in the same $thi at the same time? If not, why are there two variables?

    Because now, it's far from clear to me when to use $thi, and when to use $job, for example, why I have to use $thi->err and not $job->err.

      Inside the thread I have a list of jobs, I can create many jobs, have the reference for them, than wait to have them executed. But note that this jobs will be executed one by one.

      Graciliano M. P.
      "Creativity is the expression of liberty".

Re: Isolating dynamically loaded modules with Thread::Isolate.
by perrin (Chancellor) on Jan 30, 2005 at 14:51 UTC
    The simpler answer is to use processes instead of threads. On Linux and other unixy OSes, you can fork multiple processes and most of the memory will be shared by the OS using copy-on-write. Perl threads don't do that, so they use much more memory.
      Preach it.

      It's completely mystifying to me that people are using Perl threads for real work on UNIX systems. Much of the promise of threads - lower memory footprint, faster spawn and join, easier management - are absent from Perl's implementation. All that's left is easier communication between tasks which doesn't seem like much of a gain compared to the drawbacks.

      -sam

        I you don't need inter-task communications, then I agree, but the biggest problem people have with using threads is trying to use them in the same way that forks are used. This is true of whether using iThreads or pthreads, or most other forms of threading.

        Just as Perl requires the adoption of a different way of working to C if you are to realise the best from it, so once you have relatively easy inter-task communications, a different way of working and thinking is called for.

        And there are other platforms than the unix-like world.


        Examine what is said, not who speaks.
        Silence betokens consent.
        Love the truth but pardon error.
        I consider the easier communication to actually be a drawback of threads. More opportunities for accidental race conditions!
        A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Isolating dynamically loaded modules with Thread::Isolate.
by BrowserUk (Patriarch) on Jan 29, 2005 at 22:43 UTC

    Have you ever looked at the async call in threads?


    Examine what is said, not who speaks.
    Silence betokens consent.
    Love the truth but pardon error.
      I say to you: "Have you ever looked at the async call in threads?"

      async(&;@) is just an alias to:

      threads->new( sub {...} );
      What we are doing here is a thread that can be used externally, I can call things from outside inside this thread. Now a normal thread in Perl is just a sub that is called, and what happens after this we don't know or control.

      In a normal Perl thread, after create it, we can't say externally to load some modules or to call functions inside it, and this is the point here!

      Graciliano M. P.
      "Creativity is the expression of liberty".

        What we are doing here is a thread that can be used externally

        I do not understand this statement?

        , I can call things from outside inside this thread.

        Nor this one? If I call into one thread, from another thread, does the calling thread block until the called thread completes? If so, the what is the thread doing for me? If not, how does this differ from starting the thread and then calling join to retrieve the results at some later point?

        I've downloaded the module, read the pod and looked through the code, and I am a loss to understand as to what this module does beyond using eval to defer loading of modules until runtime? Which can be achieved using require and import.

        I see lots of API, and Storable used to freeze something and then thaw it. I see lots of state that replicates state internal to threads.pm.

        What I do not see is why I would use it?

        This isn't critisism of the module--though it maybe of the documentation a little. It is just me trying to understand why I might use it. The examples do not make it clear to me what the module gets me?


        Examine what is said, not who speaks.
        Silence betokens consent.
        Love the truth but pardon error.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlmeditation [id://426297]
Approved by davido
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chilling in the Monastery: (2)
As of 2024-03-19 05:23 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found