FileHandles and threads

[sodul]

Hi Monks,

I have a Perl script that need to be threaded (handles http requests) and also need to read and write in a forked subprocess. The subprocess has a very slow start time and I call it using the open2 function.

My problem is that I cannot find a way to share the filehandles between my threads.

I can do:

use FileHandle;
use threads::shared;
use vars qw ($in, $out);

share $in; share $out;

# but if I try to put a handle in the variables "share" will complain
# that the type is invalid
$in =new FileHandle;   # error

So is it possible to open/create a filehandle in a thread and share/reuseit in a different thread ? Reopening with a new file handle is not an option because of the slow loading of the subprocess.

Been stuck for days on trying to figure out a way to make it work.

I'm looking forward to your replies,

Regards.

Stephane

[BrowserUk]

It is possible to share a filehandle from multiple threads (but there may be caveats). The problem is how to pass the handle (an object in Perl's terms) to the thread that is going to perform the further processing as iTthreads won't allow you to share objects.

The solution is to pass the fileno associated with the filehandle and then "dup" the handle within the thread using the syntax  open my $newhandle, "<&=$fileno" ... Note:That is "<&=$fileno" (an alias file descriptor) not "<&$fileno" (a duplicate file descriptor) (see perlopentut for details). Note: Although this uses the open built-in syntax, it is not reopening the file, mearly duplicating the internal control structures required for accessing the existing open filehandle.

This incomplete demo, open files in the main thread, prints the first 10 lines before passing the fileno for that filehandle to a newly created thread that prints the rest of the file, closes it and dies:

#! perl -slw
use strict;
use threads;
use threads::shared;

sub thread{
    my( $fileno ) = @_;
    open my $fh, "<&=$fileno" or warn $! and die;
    printf "%d:%s", threads->self->tid, $_ while defined( $_ = <$fh> )
+;
    close $fh;
}

for my $file ( map{ glob $_ } @ARGV ) {
    open my $fh, '<', $file or warn "$file : $!" and next;
    printf "From main: %s", scalar <$fh> for 1 .. 10;
    printf "Fileno:%d\n", fileno $fh;
    threads->create( \&thread, fileno( $fh ) )->detach;
    printf 'paused:';<STDIN>;
}

This seems to work for simple filehandles and sockets, but I see no reason (but have not verified) why it should not also work for those created using Filehandle, just use the fileno method.

The one caveat I am aware of is that I have occasionally seen strange effects when trying this with files opened for read-write access, but it is transient and usually "goes away". That is to say, I've only seen it occasionally and it has always appeared to fix itself when other bugs where resolved, but I have done very limited testing with this. If you wish to pursue this, you will be a pioneer. Feedback welcomed.

[sodul]

Thanks a lot this technique is working very well.

I just need to be careful that the original $fh is not destroyed, which require to use a "global" variable.

You made my day.

- Stephane

[zentara]

Thanks for that code example! Passing a filehandle through a thread shared variable was something I've been searching for for awhile. Just for demonstration purposes, it can go the other way too.....create the filehandle in the thread, and read it in main.
Read more... (2 kB)

[BrowserUk]

I first played with this a couple of years ago. I first mentioned it here about a year ago in Re^3: Passing globs between threads.

By using a pool of threads and passing the filenos via a queue, it is possible to efficiently emulate all the usual forms of forking server architecture.

[Anonymous Monk]

hello

i have used the code and comment, because my problem was a little similar. i need to access to a big file (6Go / 12M protein records) to search for fuzzy similary

so i give my solution, it's a good start and maybe it will be useful to someone

regards

yovoa

 #! perl -slw
use strict;
use warnings;
use threads;
use Thread::Queue;


my $file="/path_to_the_file" ;
our $file_handler ;
our $ref_file_handler ;
my ( $process_1 , $process_2 , $process_3 ) ;
my $flag_on_file = Thread::Queue->new ;


# main
open $file_handler, '<', $file or warn "$file : $!" and die;
$flag_on_file->enqueue(fileno($file_handler));

# threads creation
$process_1 = threads->new( \&my_sub );
$process_2 = threads->new( \&my_sub );
$process_3 = threads->new( \&my_sub );

# waiting for all threads to stop before exit
$process_1->join;
$process_2->join;
$process_3->join;

# end main


# sub
sub my_sub
{
    my( $fileno ) = $flag_on_file->dequeue ;
    open $ref_file_handler, "<&=$fileno" or warn $! and die;
for ( 1..10 )
    {
    # do something with the file
    # exemple
    print threads->self->tid.": ".scalar <$ref_file_handler> ;
    }
    close $ref_file_handler;
    $flag_on_file->enqueue(fileno($file_handler));
    # do something with data
}


exit;