Re^3: How to split file for threading?

[BrowserUk]

File is remote, but I think everything I do is remote since I have to log in to the server

The question is whether the file is local or remote to the processor running the program; not the human who initiates it.

You could try this, but it is doubtful if it will be any faster when using multiple threads than just one unless the file is on a local, fast, SSD:

#! perl -slw
use strict;
use threads;

sub worker {
    my( $filename, $target, $start, $end ) = @_;
    open my $fh, '<', $filename or die $!;

    seek $fh, $start, 0;
    <$fh> if $start > 0; ## discard first partial line

    my $count = 0;
    1+index( <$fh>, $target ) and ++$count while tell( $fh ) < $end;

    return $count;
}

our $T //= 4;

my( $filename, $target ) = @ARGV;
my $fsize = -s $filename;
my $chunksize = int( $fsize / $T );
my @chunks = map{ $_ * $chunksize } 0 .. $T-1;
push @chunks, $fsize;

my @threads = map{
    threads->new( \&worker, $filename, $target, $chunks[ $_ ], $chunks
+[ $_+1 ] )
} 0 .. $T-1;

my $total = 0;
$total += $_->join for @threads;

print "Found $total '$target' lines";

Usage:

thisScript.pl -T=n theFile.txt "the string"

Note:The count is printed to stdout. Redirect it if you need it in a file.