Re^8: Forks, Pipes and Exec (file descriptors)

[BrowserUk]

The last problem I seem to be fighting is that when I run the code to set a pipe as nonblocking, it isn't really nonblocking. I seem to get a chunk of data every 15 seconds or so on Win2k3 using this code:

That problem has nothing to do with blocking. You are simply Suffering from buffering. The addition of $|++; to the top of yes.pl and you will get your output once per second.

However, if the connected process is not a perl script, but some executable that doesn't disable buffering, then there may be nothing that you can do about this unless you can modify and re-build that executable. If you were using the underlying OS api calls to create the pipe, then you can disable buffering from either end, but Perl doesn't expose that functionality.

If there isn't anything on the pipe that doesn't matter, I just continue and service the next request that may/may not be waiting. Select is never used so I didn't see this as an issue.

Hm. Unless I am misunderstanding you, it does matter. If you try to read from a pipe and there is nothing available, then the read will block until there is. Even if the pipe is non-blocking. Which means you won't be able to "continue and service the next request" once you enter a readstate on one pipe until something becomes available on that pipe.

Setting the pipe non-blocking allows you to use select to discover when the is something to be read and only enter the read when you know it will be satisfied immediately. But, I said above, there is no way to set an anonymous pipe non-blocking on win32.

---

With respect to an alternative way to code yout application so that it will work on Win32. I'm afraid I still find your descriiption insuffucuently clear to suggest anything. There are several bit of this latest post that leave me confused. For example,

• when you say

  "I'm using the non-blocking because I have a thread that executes..."

  Do you mean you are already using threads--explicitly? Or are you using (pseudo-)processes via fork?

• And

  " is to provide an instance of the module that can start another process/thread/etc."

  Again, which is it? Processes? Threads? Both? And what is "etc."

  Please don't take that as pendantry. Too often people will use these terms interchangably, but it is important to distinguish between them.

• The two processes will communicate with each other in a command/result xml format (all commands come from the instantiated module and are sent to the child process). The child process is the responsible for servicing those requests which includes starting up additional cli programs and buffering their output. The buffer will be regularly checked for errors and when the parent process requests the errors, they are passed back up.

  That implies you are talking about bi-directional communications via pipes--Expect style. I don't think anyone has got that to work from Perl on Win32.

I hope that explains things.

Sorry, but no it doesn't. At this point, the overall architecture of your application is about as clear as mud to me. It involves child processes, and pipes, and sending commands to the children, and getting replies back, and running multiple of these concurrently, but I have no overview.

And quite how you are achieving this on linux without using select is beyond me?

If you have code that works on Linux and don't mind letting me see it--either here or via email--that would certainly be the quickest way for me to understand what you are doing and perhaps be able to suggest how to make it work on Win32.

From what I've understood of what you've said POE certainly sounds like an option, though as you imply it would probably mean re-writing everything you have and involve a pretty steep learning curve. I'm also not sure how portable POE code is to Win32.

[diabelek]

I can definitely change the buffering on the other application and that helps a bit.

As for the non-blocking, I thought non-blocking allowed a read to complete even when there was no data. If there was no data, I thought either EOF or EAGAIN or something was set. The example above can be changed to this to get it working in linux and loop as fast as possible.

use IO::Handle;
use Timer::HiRes qw(usleep);

print "before the pipe\n";

sub getpipe {
   my $hash2 = shift;
   print( "starting yes.pl " . time() . "\n" );
   $hash2->{pid} = open( $hash2->{stdout}, "/usr/bin/perl yes.pl |") o
+r die;
   $hash2->{buffer} = "";
   $hash2->{stdout}->blocking(0);
   sleep( 1 );
   return $hash2->{pid};
}

my $hash = {};
getpipe($hash);
my $fh = $hash->{stdout};
print( "pipe created\n" );

my $count = 0;
while( 1 ) {
   my $data = <$fh>;
   if( $data != "" ) {
      print "got from pipe: $data";
   }
   print( "." );
   usleep( 10000 );
}
print( "done\n" );

I will try to post the script externally and provide a link so you understand what I'm trying to do a little better. To answer a couple of your questions about threads/processes, I'm really using both. I start the thread/process by calling fork. So in Linux, I understand that I'm using the true fork command. In Windows, I'm using the pseudo fork that is really thread when you get to the bottom of things. Which one I use doesn't matter. I just need a separate branch of execution so the parent script can continue doing its normal thing while my branch of execution handles the external processes. Thats probably an ignorant statement but that's why I'm posting :)

I've found another perl script that sounds like what I'm doing: logtail. I believe the basic idea is the same. I need to review the code more to see what they have done.

[BrowserUk]

I've found another perl script that sounds like what I'm doing: logtail.

Here's a simplified and somewhat crude equivalent that uses threads and should run anywhere you have a threaded-perl and a tail command. Maybe it'll be useful to you.

#! perl -slw
use strict;
use threads ( stack_size => 4096 );
use threads::shared;
use Thread::Queue;

$|++;

our $VERBOSE :shared;
our $REMOTE :shared;
my $stop :shared = 0; ## Set true to terminate threads

my @logs = map glob, @ARGV; ## expand wildcards
my $Q = new Thread::Queue;

## One trhead per log file
threads->create( \&tail, $Q, $_, 1 )->detach for @logs;

my $remote; ## Remote watcher socket
if( $REMOTE ) {
    require IO::Socket;
    $remote = IO::Socket::INET->new( $REMOTE )
        or warn "Couldn't connect to $REMOTE : $!, $^E";
    print $remote "Hi there, Got your ears on?";
}

## Thread to monitor the Q, print locally and/or forward to remote
my $relay = async {
    for( 1 .. @logs ) { ## Waits for all tals to terminate
        while( my $line = $Q->dequeue ){
            chomp $line;
            print $line if $VERBOSE;
            print $remote $line if $remote;
        }
    }
};

## Local command loop
while( <STDIN> ) {
    my( $command, $value ) = split;

    if( $command =~ m[^(END|QUIT)]i ) {
        $stop = 1;
        warn "Quiting...\n";
        $relay->join;
        exit 0;
    }

    if( $command =~ m[^VERBOSE]i ) {
        $VERBOSE = $value;
        print "VERBOSE set to $value";
    }

    elsif( $command eq 'qs' ) {
        print $Q->pending;
    }

    else {
        print "Unrecognised command: $command";
    }
}

## Tail threads
sub tail {
    print threads->tid, ' : ', threads->self->get_stack_size;
    my( $Q, $path, $seconds ) = @_;
    $seconds = 1 unless $seconds;
    my $pid = open my $log, "tail -Fs $seconds $path |"
        or die $@;
    printf "Thread %d Following $path\n", threads->tid;
    $Q->enqueue( $_ ) while not $stop and defined( $_ = <$log> );
    kill 3, $pid;
    close $log;
    $Q->enqueue( undef );
}

It happily follows 100 logs, simultaneously logging them local and transmitting them to a remote watcher. You can enter a few commands at the local console to turn local verbose on and off, monitor the size of the queue, and quit:

c:\test>logwatch.pl -REMOTE=localhost:35007 log\log*.txt
Thread 2 Following log\log0002.txt
Thread 1 Following log\log0001.txt
Thread 3 Following log\log0003.txt
Thread 4 Following log\log0004.txt
Thread 5 Following log\log0005.txt
Thread 6 Following log\log0006.txt
Thread 7 Following log\log0007.txt
Thread 8 Following log\log0008.txt
Thread 9 Following log\log0009.txt
Thread 10 Following log\log0010.txt
verbose 1
VERBOSE set to 1
Sat Nov  8 15:09:45.575 2008 : Message from log 6
Sat Nov  8 15:09:45.684 2008 : Message from log 0
Sat Nov  8 15:09:45.701 2008 : Message from log 3
Sat Nov  8 15:09:45.794 2008 : Message from log 3
...
ver
...
Sat Nov  8 15:09:48.279 2008 : Message from log 9
...
bose 0
...
Sat Nov  8 15:09:48.309 2008 : Message from log 1
Sat Nov  8 15:09:48.419 2008 : Message from log 9

VERBOSE set to 0
qs
0
quit
Quiting...

[diabelek]

I've switched to using sockets and threads but I've run into an interesting problem. On Windows only (shocker), I am unable to start a thread inside of another thread. I'm unclear on why it stops since depending on where I place the thread->new() it works or doesn't work. I've posted the module code here.

You'll notice in ToolHandler the while loops grabs a command and executes it and then tries to read the queue for the other processes that are running the IO tool. If I place the thread->new() before the _CommandHandler call, it fails to start the new thread and just hangs indefintely.

I'm working on putting together a simple version of the module as a script to see if simplifying it changes anything but if there is any advice on the module it would be appreciated.