IPC3 buffer limit problem

[zentara]

I was toying with using IPC3 to run bc as an educational example. See this nodeIPC3 buffer problemAnyways, I came up with some code to beat the 4k buffer limit, to give unlimited bc output. It's a hard to solve problem, do a groups.google search for "IPC perl buffer size limit". Anyways, here is what I came up with. All comments and criticisms welcome. It's how I learn :-) The basic trick I use is to use a do loop to keep reading the buffer until the buffer is less than full. Then I display the concantenated output string. The 4k buffer is 4060 bytes on my linux machine.

#!/usr/bin/perl
use warnings;
use strict;
use IPC::Open3;
require 'sys/ioctl.ph';

# I tested this with  
#  "q @ 2" to generate errors 
# "123^12345" to generate big output 
# "123^23456" to generate huge output 

#interface to "bc" calculator 
my $pid = open3(\*WRITE, \*READ,\*ERROR,"bc");
            #if \*ERROR is false, STDERR is sent to STDOUT  

while(1){

my($error,$answer,$rsize,$esize,$errortot,$answertot)=('','',0,0,'',''
+);

print "Enter expression for bc, i.e. 2 + 2\n";
chomp(my $query = <STDIN>);
if(length($query) == 0){next} #removes empty input 

#send query to bc 
print WRITE "$query\n";

# this do loop waits and eliminates need for a delay 
# to wait for bc to  output 
# It is NOT limited to 4060 bytes from bc, it keeps reading 
# until a semi-full buffer is sent, which signals end of output 

  do {
      #see which filehandles have output from perldoc -q filehandle 
      $esize = pack("L", 0);
      ioctl(\*ERROR, FIONREAD(), $esize) or die "Couldn't call ioctl: 
+$!\n";
      $esize = unpack("L", $esize);
      print "esize-> $esize\n" unless ($esize < 1);

      $rsize = pack("L", 0);
      ioctl(\*READ, FIONREAD(), $rsize) or die "Couldn't call ioctl: $
+!\n";
      $rsize = unpack("L", $rsize);
      print "rsize-> $rsize\n" unless ($rsize <1);

     #get the output from bc 
     if($esize > 0){sysread(ERROR,$error,$esize); $errortot = $errorto
+t.$error}
     if($rsize > 0){sysread(READ,$answer,$rsize); $answertot = $answer
+tot.$answer}
    } until(($esize > 0)or(($rsize > 0)and($rsize < 4060)));


if(length($errortot) > 0){ print "\e[1;31m ERROR-> $errortot \e[0m \n"
+}
if(length($answertot) > 0){print "Output->$query = $answertot\n"}

}

[merlyn]

By providing tiny inputs, you're not hitting the real problem. If you need to pump more than 4K into the input buffer but can't start reading the output until after that, and the output hits 4K, you're toast.

Certainly, if you could use async or pseudo-async handlers, you've got it made. See the way that POE deals with child running processes, especially POE::Wheel::Run.

[zentara]

Merlyn said: If you need to pump more than 4K into the input buffer but can't start reading the output until after that, and the output hits 4K, you're toast.

Yeah, I realized that after I posted it. The "edge case" of where the last buffer filled is 0 or 4060 is a problem. More thinking is needed.

[zentara]

Update: Jan-03-2006

just tried to run this script again on my newer linux system with a 2.6.x kernel and Perl 5.8.7. Anyways, there seems to be a difference in the way pipes work, and the above script no longer works. Below is another version that does work. It includes an output-to-file, so you can compare the output to bc running in a terminal.

#!/usr/bin/perl
use warnings;
use strict;
use IPC::Open3;
require 'sys/ioctl.ph';
use FileHandle; # for testing output to compare to 
                # bc's output, needed for autoflush 

$| = 1;

#to write to a file to compare to bc's output 
open (OUT, "> $0.out") or die "$!\n";
OUT->autoflush(1);

# I tested this with  
#  "q @ 2" to generate errors 
# "123^12345" to generate big output 
# "123^23456" to generate huge output 

#interface to "bc" calculator 
my $pid = open3(\*WRITE, \*READ,\*ERROR,"bc");
            #if \*ERROR is false, STDERR is sent to STDOUT  

while(1){

my($error,$answer,$rsize,$esize,$errortot,$answertot)=('','',0,0,'',''
+);

print "Enter expression for bc, i.e. 2 + 2\n";
chomp(my $query = <STDIN>);
if(length($query) == 0){next} #removes empty input 

#send query to bc 
print WRITE "$query\n";

#this do loop waits and eliminates need for a delay to let bc output 
#It is NOT limited to 4060 bytes from bc, it keeps reading 
#until a semi-full buffer is sent, which signals end of output 
  do {
      #see which filehandles have output from perldoc -q filehandle 
      $esize = pack("L", 0);
      ioctl(\*ERROR, FIONREAD(), $esize) or die "Couldn't call ioctl: 
+$!\n";
      $esize = unpack("L", $esize);
      print "esize-> $esize\n" unless ($esize < 1);

      $rsize = pack("L", 0);
      ioctl(\*READ, FIONREAD(), $rsize) or die "Couldn't call ioctl: $
+!\n";
      $rsize = unpack("L", $rsize);
      print "rsize-> $rsize\n" unless ($rsize <1);

     #get the output from bc 
     if($esize > 0){sysread(ERROR,$error,$esize); $errortot = $errorto
+t.$error}

#    needed this line on older linuxes and Perl, the pipes worked diff
+erent 
#     if($rsize > 0){sysread(READ,$answer,$rsize); $answertot = $answe
+rtot.$answer} 

# on newer kernels 2.6.x and Perl 5.8.7, this works 
     if($rsize > 0){sysread(READ,$answer,4096); $answertot = $answerto
+t.$answer}

  } until( ($esize > 0)  or ( ($rsize > 0) and ($rsize < 4096) )  );

if(length($errortot) > 0){ print "\e[1;31m ERROR-> $errortot \e[0m \n"
+}
if(length($answertot) > 0){print "Output->$query = $answertot\n" }

print OUT $answertot;
}

close OUT;

waitpid($pid, 1);
# It is important to waitpid on your child process,  
# otherwise zombies could be created.