Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

IPC3 buffer limit problem

by zentara (Archbishop)
on Oct 19, 2003 at 20:46 UTC ( #300420=snippet: print w/ replies, xml ) Need Help??

Description: I was toying with using IPC3 to run bc as an educational example. See this nodeIPC3 buffer problemAnyways, I came up with some code to beat the 4k buffer limit, to give unlimited bc output. It's a hard to solve problem, do a groups.google search for "IPC perl buffer size limit". Anyways, here is what I came up with. All comments and criticisms welcome. It's how I learn :-) The basic trick I use is to use a do loop to keep reading the buffer until the buffer is less than full. Then I display the concantenated output string. The 4k buffer is 4060 bytes on my linux machine.
#!/usr/bin/perl
use warnings;
use strict;
use IPC::Open3;
require 'sys/ioctl.ph';

# I tested this with  
#  "q @ 2" to generate errors 
# "123^12345" to generate big output 
# "123^23456" to generate huge output 

#interface to "bc" calculator 
my $pid = open3(\*WRITE, \*READ,\*ERROR,"bc");
            #if \*ERROR is false, STDERR is sent to STDOUT  

while(1){

my($error,$answer,$rsize,$esize,$errortot,$answertot)=('','',0,0,'',''
+);

print "Enter expression for bc, i.e. 2 + 2\n";
chomp(my $query = <STDIN>);
if(length($query) == 0){next} #removes empty input 

#send query to bc 
print WRITE "$query\n";

# this do loop waits and eliminates need for a delay 
# to wait for bc to  output 
# It is NOT limited to 4060 bytes from bc, it keeps reading 
# until a semi-full buffer is sent, which signals end of output 

  do {
      #see which filehandles have output from perldoc -q filehandle 
      $esize = pack("L", 0);
      ioctl(\*ERROR, FIONREAD(), $esize) or die "Couldn't call ioctl: 
+$!\n";
      $esize = unpack("L", $esize);
      print "esize-> $esize\n" unless ($esize < 1);

      $rsize = pack("L", 0);
      ioctl(\*READ, FIONREAD(), $rsize) or die "Couldn't call ioctl: $
+!\n";
      $rsize = unpack("L", $rsize);
      print "rsize-> $rsize\n" unless ($rsize <1);

     #get the output from bc 
     if($esize > 0){sysread(ERROR,$error,$esize); $errortot = $errorto
+t.$error}
     if($rsize > 0){sysread(READ,$answer,$rsize); $answertot = $answer
+tot.$answer}
    } until(($esize > 0)or(($rsize > 0)and($rsize < 4060)));


if(length($errortot) > 0){ print "\e[1;31m ERROR-> $errortot \e[0m \n"
+}
if(length($answertot) > 0){print "Output->$query = $answertot\n"}

}

Comment on IPC3 buffer limit problem
Download Code
•Re: IPC3 buffer limit problem
by merlyn (Sage) on Oct 19, 2003 at 23:05 UTC
    By providing tiny inputs, you're not hitting the real problem. If you need to pump more than 4K into the input buffer but can't start reading the output until after that, and the output hits 4K, you're toast.

    Certainly, if you could use async or pseudo-async handlers, you've got it made. See the way that POE deals with child running processes, especially POE::Wheel::Run.

    -- Randal L. Schwartz, Perl hacker
    Be sure to read my standard disclaimer if this is a reply.

Re: IPC3 buffer limit problem
by zentara (Archbishop) on Oct 20, 2003 at 13:58 UTC
    Merlyn said: If you need to pump more than 4K into the input buffer but can't start reading the output until after that, and the output hits 4K, you're toast.

    Yeah, I realized that after I posted it. The "edge case" of where the last buffer filled is 0 or 4060 is a problem. More thinking is needed.

Re: IPC3 buffer limit problem
by zentara (Archbishop) on Jan 03, 2006 at 15:12 UTC
    Update: Jan-03-2006

    just tried to run this script again on my newer linux system with a 2.6.x kernel and Perl 5.8.7. Anyways, there seems to be a difference in the way pipes work, and the above script no longer works. Below is another version that does work. It includes an output-to-file, so you can compare the output to bc running in a terminal.

    #!/usr/bin/perl use warnings; use strict; use IPC::Open3; require 'sys/ioctl.ph'; use FileHandle; # for testing output to compare to # bc's output, needed for autoflush $| = 1; #to write to a file to compare to bc's output open (OUT, "> $0.out") or die "$!\n"; OUT->autoflush(1); # I tested this with # "q @ 2" to generate errors # "123^12345" to generate big output # "123^23456" to generate huge output #interface to "bc" calculator my $pid = open3(\*WRITE, \*READ,\*ERROR,"bc"); #if \*ERROR is false, STDERR is sent to STDOUT while(1){ my($error,$answer,$rsize,$esize,$errortot,$answertot)=('','',0,0,'','' +); print "Enter expression for bc, i.e. 2 + 2\n"; chomp(my $query = <STDIN>); if(length($query) == 0){next} #removes empty input #send query to bc print WRITE "$query\n"; #this do loop waits and eliminates need for a delay to let bc output #It is NOT limited to 4060 bytes from bc, it keeps reading #until a semi-full buffer is sent, which signals end of output do { #see which filehandles have output from perldoc -q filehandle $esize = pack("L", 0); ioctl(\*ERROR, FIONREAD(), $esize) or die "Couldn't call ioctl: +$!\n"; $esize = unpack("L", $esize); print "esize-> $esize\n" unless ($esize < 1); $rsize = pack("L", 0); ioctl(\*READ, FIONREAD(), $rsize) or die "Couldn't call ioctl: $ +!\n"; $rsize = unpack("L", $rsize); print "rsize-> $rsize\n" unless ($rsize <1); #get the output from bc if($esize > 0){sysread(ERROR,$error,$esize); $errortot = $errorto +t.$error} # needed this line on older linuxes and Perl, the pipes worked diff +erent # if($rsize > 0){sysread(READ,$answer,$rsize); $answertot = $answe +rtot.$answer} # on newer kernels 2.6.x and Perl 5.8.7, this works if($rsize > 0){sysread(READ,$answer,4096); $answertot = $answerto +t.$answer} } until( ($esize > 0) or ( ($rsize > 0) and ($rsize < 4096) ) ); if(length($errortot) > 0){ print "\e[1;31m ERROR-> $errortot \e[0m \n" +} if(length($answertot) > 0){print "Output->$query = $answertot\n" } print OUT $answertot; } close OUT; waitpid($pid, 1); # It is important to waitpid on your child process, # otherwise zombies could be created.

    I'm not really a human, but I play one on earth. flash japh

Back to Snippets Section

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: snippet [id://300420]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (11)
As of 2014-10-01 16:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    What is your favourite meta-syntactic variable name?














    Results (29 votes), past polls