Wrap around an interactive python script

[Kirsle]

Fellow monks, I have need of your wisdom.

I'm trying to write a Perl script that wraps around an interactive Python script (it's the "upload.py" for Google's code review service). My script needs to get the output given to the user so that it can parse out the issue URL generated by upload.py.

I was thinking I would just do something like this:

my $output = `/script/upload.py @ARGV`;

I was expecting that this would run upload.py and print what upload.py prints to the user's terminal so they can see it, and when they're prompted to fill out information, it would pass their input through to upload.py directly.

But the backticks don't let STDOUT sneak through to the terminal, so then I tried using system() with "tee" to copy the STDOUT to a file which my script could read after:

system("/script/upload.py @ARGV | tee temp.tee");
my $output = `cat temp.tee`; # not really, did a read() instead

But this didn't work either. No output from upload.py was given on my terminal; it was obviously hanging, waiting for input, but didn't give any output unless I made the upload.py exit normally (i.e. answering "no" when it prompted me about svn files that aren't managed by svn at the beginning). It may be a line buffering issue in upload.py; how could I force autoflush on it if this is the case?

[ahmad]

You can do this through open pipe, but you might run into some buffering problems.

open(LOG,">>/path/to/log");
open(SYS,"/path/to/python/script.py |");
while (<SYS>) {
      print $_;
      print LOG $_;
}
close(SYS);
close(LOG);

This way you can print to screen and to your log file

[rastoboy]

I've recently started using IPC::Open3::open3 for InterProcess Communication type purposes and haven't looked back. Makes it easy to interact with another application by just reading and writing to filehandles, very slick and more intuitive (at least for me, anyway).

But I'm still quite the amateur and may be missing some subtleties (or outright facts).

[ikegami]

While open3 does a lot (incl propagating exec errors to the parent), it's still a low level module. It's doesn't help you read from the child's STDOUT and its STDERR without combining them, a situation prone to deadlocks. IPC::Run3 and IPC::Run are higher-level solutions.

[ikegami]

Expect is designed to work with interactive programs. One of the things it does is use a pseudo tty. This "fools" most programs into disabling output buffering.

[cdarke]

It may be a line buffering issue in upload.py; how could I force autoflush on it if this is the case?

If you set $| = 1; in Perl, and assuming that STDOUT is the default file handle, then that should be inherited by the python program. Otherwise (forgive me monks for I have sinned) in Python you can

sys.sysout.flush()

although I admit that is not an autoflush.

[afoken]

I don't think so. Autoflushing output happens in the C runtime library (libc/glibc) or even above that, and the auto flush flag is also a thing of the C runtime library or the interpreter or application above that. It is not part of the O/S information about the file handle, and so it does not survive the exec() system call. You have to enable autoflushing in Python (or any other application you start).

The flag does survive a fork(), like any other simple variable does.

Alexander