Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

IPC::Run::run and buffering

by rovf (Priest)
on Jun 14, 2012 at 08:44 UTC ( #976146=perlquestion: print w/ replies, xml ) Need Help??
rovf has asked for the wisdom of the Perl Monks concerning the following question:

I have an application, which uses IPC::Run in the following way:

IPC::Run::run([$cmd,@args],'>>',$outpath,'2>&1')
I have no information about $cmd - can be anything, and it is not on my control. Now my question:

Under the assumption that $cmd does not do unbuffered output, and doesn't fiddle around with buffering by itself (think of: just doing C style fwrite calls), is there a way I can control the buffer size (i.e. make it larger) of the pipe from the Perl side?

The code is currently executed on Windows and Linux. I understand that if there is at all a solution to my problem, it probably will look different on both platforms. This is OK. I'm not very optimistic, that a solution to this problem exist, but I don't know enough about the operating system internals.

Another idea which came to my mind, would be to replace the "IPC::Run::run" by an open(my $pipe,"$cmd ...|") and read from this pipe to my application, which then writes the output in larger chunks. This should work at least on Linux (possibly not Windows, because Windows does not, AFIK, support real pipes and needs to simulate them using files). However, if there were a solution of just increasing the buffer size for the output redirection, I would prefer that.

UPDATE:

I would like to sketch a possible solution - maybe someone can see whether or not it should work: I replace the call to IPC::Run::run by a call to IPC::Run::start and collect the output via a function:

my $harness=start(\@cmd,\'',\&f,\&f)
The function buffers the resulting lines, and if the number of characters exceeds a certain amount, it "flushes" the buffer by writing it to the output file. In order to bypass system-internal buffering, the output file is opened for writing using the :unix layer, and the data is written using syswrite instead of print. This is done in order to bypass stdio buffering. Of course this means that on Windows, I have to manually translate \n into 0x0d.0x0a manually.

Does this sound reasonable? Is there a simpler way to do it?
-- 
Ronald Fischer <ynnor@mm.st>

Comment on IPC::Run::run and buffering
Select or Download Code
Re: IPC::Run::run and buffering
by Anonymous Monk on Jun 14, 2012 at 09:38 UTC

    You're under the impression you need to increase the buffer size?

    Try to write a program that triggers whatever problem you think you're encountering, like this program

    my $cmd = $^X; my @args = ( qw' -l -e ', 'print $_ for 1 .. 10_000;' );
Re: IPC::Run::run and buffering
by zentara (Archbishop) on Jun 14, 2012 at 10:58 UTC
      I don't know why you need to increase buffer size
      It happens that several copies of this program are running simultaneously, and the all write to files on the same host. With the default buffer size, this causes too many I/O accesses, sometimes causing long delays until the actual writing operation has been completed. I would like to try weather increasing the buffer size, hence decreasing the number of simultaneous writes, would give a better performance.

      I checked the link you have provided, but it doesn't give much insight into my problem (i.e. Perl version greater than 5.8, and buffered writing (not reading)).
      BTW, I've updated my node. If you have time, it would be great if you could have a look at it again.

      -- 
      Ronald Fischer <ynnor@mm.st>
Re: IPC::Run::run and buffering
by BrowserUk (Pope) on Jun 14, 2012 at 14:39 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://976146]
Approved by marto
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (7)
As of 2014-10-30 23:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (211 votes), past polls