Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

FileHandles and threads

by sodul (Initiate)
on Sep 21, 2005 at 12:25 UTC ( #493754=perlquestion: print w/ replies, xml ) Need Help??
sodul has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I have a Perl script that need to be threaded (handles http requests) and also need to read and write in a forked subprocess. The subprocess has a very slow start time and I call it using the open2 function.

My problem is that I cannot find a way to share the filehandles between my threads.

I can do:

use FileHandle;
use threads::shared;
use vars qw ($in, $out);

share $in; share $out;

# but if I try to put a handle in the variables "share" will complain
# that the type is invalid
$in =new FileHandle;   # error

So is it possible to open/create a filehandle in a thread and share/reuseit in a different thread ? Reopening with a new file handle is not an option because of the slow loading of the subprocess.

Been stuck for days on trying to figure out a way to make it work.

I'm looking forward to your replies,

Regards.

Stephane

Comment on FileHandles and threads
Re: FileHandles and threads
by BrowserUk (Pope) on Sep 21, 2005 at 14:08 UTC

    It is possible to share a filehandle from multiple threads (but there may be caveats). The problem is how to pass the handle (an object in Perl's terms) to the thread that is going to perform the further processing as iTthreads won't allow you to share objects.

    The solution is to pass the fileno associated with the filehandle and then "dup" the handle within the thread using the syntax  open my $newhandle, "<&=$fileno" ... Note:That is "<&=$fileno" (an alias file descriptor) not "<&$fileno" (a duplicate file descriptor) (see perlopentut for details). Note: Although this uses the open built-in syntax, it is not reopening the file, mearly duplicating the internal control structures required for accessing the existing open filehandle.

    This incomplete demo, open files in the main thread, prints the first 10 lines before passing the fileno for that filehandle to a newly created thread that prints the rest of the file, closes it and dies:

    #! perl -slw use strict; use threads; use threads::shared; sub thread{ my( $fileno ) = @_; open my $fh, "<&=$fileno" or warn $! and die; printf "%d:%s", threads->self->tid, $_ while defined( $_ = <$fh> ) +; close $fh; } for my $file ( map{ glob $_ } @ARGV ) { open my $fh, '<', $file or warn "$file : $!" and next; printf "From main: %s", scalar <$fh> for 1 .. 10; printf "Fileno:%d\n", fileno $fh; threads->create( \&thread, fileno( $fh ) )->detach; printf 'paused:';<STDIN>; }

    This seems to work for simple filehandles and sockets, but I see no reason (but have not verified) why it should not also work for those created using Filehandle, just use the fileno method.

    The one caveat I am aware of is that I have occasionally seen strange effects when trying this with files opened for read-write access, but it is transient and usually "goes away". That is to say, I've only seen it occasionally and it has always appeared to fix itself when other bugs where resolved, but I have done very limited testing with this. If you wish to pursue this, you will be a pioneer. Feedback welcomed.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.

      Thanks a lot this technique is working very well.

      I just need to be careful that the original $fh is not destroyed, which require to use a "global" variable.

      You made my day.

      - Stephane

      Thanks for that code example! Passing a filehandle through a thread shared variable was something I've been searching for for awhile. Just for demonstration purposes, it can go the other way too.....create the filehandle in the thread, and read it in main.

      I'm not really a human, but I play one on earth. flash japh

        I first played with this a couple of years ago. I first mentioned it here about a year ago in Re^3: Passing globs between threads.

        By using a pool of threads and passing the filenos via a queue, it is possible to efficiently emulate all the usual forms of forking server architecture.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
        "Science is about questioning the status quo. Questioning authority".
        The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.
Re: FileHandles and threads
by Anonymous Monk on Apr 20, 2013 at 18:12 UTC

    hello

    i have used the code and comment, because my problem was a little similar. i need to access to a big file (6Go / 12M protein records) to search for fuzzy similary

    so i give my solution, it's a good start and maybe it will be useful to someone

    regards

    yovoa

    #! perl -slw use strict; use warnings; use threads; use Thread::Queue; my $file="/path_to_the_file" ; our $file_handler ; our $ref_file_handler ; my ( $process_1 , $process_2 , $process_3 ) ; my $flag_on_file = Thread::Queue->new ; # main open $file_handler, '<', $file or warn "$file : $!" and die; $flag_on_file->enqueue(fileno($file_handler)); # threads creation $process_1 = threads->new( \&my_sub ); $process_2 = threads->new( \&my_sub ); $process_3 = threads->new( \&my_sub ); # waiting for all threads to stop before exit $process_1->join; $process_2->join; $process_3->join; # end main # sub sub my_sub { my( $fileno ) = $flag_on_file->dequeue ; open $ref_file_handler, "<&=$fileno" or warn $! and die; for ( 1..10 ) { # do something with the file # exemple print threads->self->tid.": ".scalar <$ref_file_handler> ; } close $ref_file_handler; $flag_on_file->enqueue(fileno($file_handler)); # do something with data } exit;

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://493754]
Approved by tbone1
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (10)
As of 2014-09-17 08:49 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (70 votes), past polls