http://www.perlmonks.org?node_id=929318

philkime has asked for the wisdom of the Perl Monks concerning the following question:

I have a module which uses an external library. I would like to catch all STDERR from calls to subs which are in the external library into a string so I can check it later.

Here is a minimal example. This prints "manual error" but does not contain the STDERR from the Text::BibTeX sub. Without the STDERR redirection, the Text::BibTeX sub error is displayed on terminal STDERR as normal.

#!/bin/perl use Text::BibTeX; my $stderr; my $bib_entry = <<'END'; @BOOK{test1, AUTHOR = {Arthur Author}, TITLE = {Arranging Arguments},, YEAR = {1935} } END close STDERR; open STDERR, '>', \$stderr; print STDERR "manual error\n"; my $entry = new Text::BibTeX::Entry $bib_entry; print $stderr; exit 0;

Replies are listed 'Best First'.
Re: STDERR going to string
by Tanktalus (Canon) on Oct 03, 2011 at 18:16 UTC

    Your basic problem is that T::BT uses XS code which in turn uses stdio instead of perl IO to print to stderr. Capturing this will become much more problematic, though not necessarily impossible.

    There may be modules that can handle this, I'm not sure. However, one for-sure way to do this is to call T::BT in a subprocess, and capture its stderr. Inside a single process, you may be able to set up threads (even Coro threads) or events with a pipe - set up your stderr to a pipe (which is handled by the C library just fine) and have an event listener/thread read it in and append it to your string manually.

    I don't think perl can do this by itself with the reference-to-a-scalar trick.

    Long term, ask the module author to use perl IO instead of stdio for their output in XS code.

      Ah, I see, thank you. I assumed it was something to do with how the library/module handled STDERR ... I'll see what I can do.
      I can get around this by redirecting to a file and reading it later. I suspect I could use Config::Tiny or File::Tee too but it seems it's a harder problem on Windows because re-opening STDERR decouples perl's STDERR from the called C library's STDERR which seems impossible to capture. If anyone has any ideas about this ... Since the issue is a library called from XS and not the XS extension itself and the fact that this library is also used apart from perl, it can't be modified to use PerlIO ...

        because re-opening STDERR decouples perl's STDERR from the called C library's STDERR

        It will probably work if you save/dup STDERR and restore it

        open my $errsave, '>&', STDERR or do { print "couldn't dup STDERR: $!"; exit 2; }; open STDERR ... ... open STDERR, '>&', $errsave or do { print "couldn't restore/dup STDERR: $!"; exit 2; };
Re: STDERR going to string
by kennethk (Abbot) on Oct 03, 2011 at 14:59 UTC
    Your described behavior sounds odd, and thus I'd really like the opportunity to replicate your behavior. If you could post a self-contained code snippet that replicates your issue, I might be able to do just that. (wrap it in <code> tags instead of <pre> tags while you are at it). The code you have posted above correctly performs the redirects for me. See I know what I mean. Why don't you?.
      I've updated the original post with a minimal example displaying the problem. It seems to work when I write to a file instead of a string, which is strange. Even stranger is that when writing to a file, the order is wrong, I see the Text::BibTeX error at the top of the file and "manual error" after this. Buffering issue? Tried this on two different machines, perl 5.14.1 on both, one cygwin, one OSX.
        Tanktalus has diagnosed your issue below, and I unfortunately have nothing useful to add.
Re: STDERR going to string
by OlegG (Monk) on Oct 03, 2011 at 15:32 UTC
    You can try to use $SIG{__WARN__} handler instead.
Re: STDERR going to string
by salva (Canon) on Oct 11, 2011 at 09:44 UTC
    There is only one truly portable and robust way to do that: redirect STDERR to a temporal file before calling into XS and read it afterwards.
    # Untested. Note also that this code will not handle special # cases as STDERR being closed, tied or attached to a file # descriptor different than 2. use autodie; use File::Temp qw(tempfile); open my $olderr, '>&STDERR'; # save STDERR my $fh = tempfile(); open STDERR, '>&', $fh; my $entry = Text::BibTeX::Entry->new($bib_entry); open STDERR, '>&', $olderr; # flush and reset STDERR seek($fh, 0, 0); my $err = do { local $/; <$fh> }; close $fh;
      I had something like this too but it doesn't work on Windows. Here is a stripped down example using Inline::C. On non-Windows (cygwin included), it outputs correctly: |ERROR| On Windows, it gives: || ERROR which shows you that the STDERR isn't being captured.
      use File::Temp qw(tempfile); use Inline C; open my $olderr, '>&STDERR'; # save STDERR my ($fh, $fn) = tempfile(); open STDERR, '>&', $fh; test_err(); open STDERR, '>&', $olderr; # reset STDERR close $fh; open my $e, '<', $fn; my $err = <$e>; print "|$err|\n"; __END__ __C__ void test_err() { (void)fprintf (stderr, "ERROR"); }
        Are you using ActiveState or Strawberry Perl? and which version?

        It seems that there is a missmatch between what Perl sees as its file descriptor 2 and the C lib file descriptor 2.

        use File::Temp qw(tempfile); use Inline 'C'; open my $olderr, '>&STDERR'; # save STDERR my ($fh, $fn) = tempfile(); open STDERR, '>&', $fh; printf "fileno STDERR: %d\n", fileno(STDERR); print STDERR "foo!"; test_err(); open STDERR, '>&', $olderr; # reset STDERR close $fh; open my $e, '<', $fn; my $err = <$e>; print "|$err|\n"; __END__ __C__ void test_err() { FILE *err = fdopen(2, "a+"); (void)fprintf (err, "ERROR"); } # here it says: # fileno STDERR: 2 # |foo!| # ERROR

        Besides that, don't trust what you get from Inline::C, the perlio.h header is included before your C code and most stdio functions are replaced by macros calling into perl own implementations (that not-too-unsurprisingly, do not work as expected either):

        update: the funny thing is that when a subprocess is launched, perl file descriptors are the ones inherited:

Re: STDERR going to string
by Anonymous Monk on Oct 12, 2011 at 09:27 UTC

    You want to use Capture::Tiny, it works on win32, it does it through fork+exec and presents a beautiful interface

    use Capture::Tiny 'capture'; my ($out, $err) = capture { print "ya"; Text::BibTeX ... system ... }; print "captured out((($out))) err((($err)))";
      Here is a simple example showing that even Capture::Tiny doesn't work in this case on Windows.
      #!/usr/bin/perl use Capture::Tiny 'capture'; use File::Temp qw(tempfile); use Inline C; my ($stdout, $stderr) = capture { test_err() }; print "|$stderr|\n"; __END__ __C__ void test_err() { (void)fprintf (stderr, "ERROR"); }
        I'm embarrassed. Someone pointed out to me that Windows doesn't flush STDERR as often as other OSES. With one extra line of C, it works on Windows and I think I can modify the external library this much ...
        #!/usr/bin/perl use Capture::Tiny 'capture'; use File::Temp qw(tempfile); use Inline C; my ($stdout, $stderr) = capture { test_err() }; print "|$stderr|\n"; __END__ __C__ void test_err() { (void)fprintf (stderr, "ERROR"); (void)fflush(stderr); }

        Here is "proof" perl (or perlxs) is stuffing stderr into a buffer
        #!/usr/bin/perl -- use Inline C => Config => BUILD_NOISY => 1, #~ FORCE_BUILD => 1, CLEAN_AFTER_BUILD => 0, ; # Inline use Inline C => <<'__C__'; void test_err() { PerlIO_printf( PerlIO_stderr(), "Inline::C PerlIO_printf => PerlIO +_stderr()(%p) is PerlIO_fileno()(%d)\n", PerlIO_stderr(), PerlIO_file +no(PerlIO_stderr()) ); fprintf (stderr, "Inline::C fprintf => PerlIO_stderr()(%p) is Perl +IO_fileno()(%d)\n", PerlIO_stderr(), PerlIO_fileno(PerlIO_stderr()) ) +; fprintf (stderr, "Inline::C fprintf => stderr(%p) is fileno(%d)\n" +, stderr, fileno(stderr) ); // above output gets eaten if you flush // fflush (stderr ); fprintf (stderr, "will somebody please eat my stderr buffer \n"); } __C__ Main( @ARGV ); print "main is over\n"; exit( 0 ); END { print "perl is over\n"; } sub Main { use FileHandle; STDERR->autoflush(1); STDOUT->autoflush(1); use File::Temp qw' tempfile '; open my $olderr, '>&STDERR'; # save STDERR my ($fh, $fn) = tempfile(); open STDERR, '>&', $fh; test_err(); open STDERR, '>&', $olderr; # reset STDERR close $fh; open my $e, '<', $fn; my $err = <$e>; print "(((($err))))\n"; } __END__

        without fflush

        ((((Inline::C PerlIO_printf => PerlIO_stderr()(99417c) is PerlIO_filen +o()(2) )))) main is over perl is over Inline::C fprintf => PerlIO_stderr()(0099417C) is PerlIO_fileno()(2) Inline::C fprintf => stderr(77C5FCC0) is fileno(2) will somebody please eat my stderr buffer

        with fflush

        ((((Inline::C PerlIO_printf => PerlIO_stderr()(99417c) is PerlIO_filen +o()(2) )))) main is over perl is over will somebody please eat my stderr buffer

        And doing the same thing WITHOUT perl , code from _dup, _dup2 adapted for mingw-gcc

        // crt_dup.c // This program uses the variable old to save // the original stdout. It then opens a new file named // DataFile and forces stdout to refer to it. Finally, it // restores stdout to its original state. // #include <io.h> #include <stdlib.h> #include <stdio.h> int main( void ) { int old; FILE *DataFile; old = _dup( 1 ); // "old" now refers to "stdout" // Note: file descriptor 1 == "stdout" if( old == -1 ) { perror( "_dup( 1 ) failure" ); exit( 1 ); } _write( old, "This goes to stdout first\n", 26 ); if( fopen_s( &DataFile, "data", "w" ) != 0 ) { puts( "Can't open file 'data'\n" ); exit( 1 ); } // stdout now refers to file "data" if( -1 == _dup2( _fileno( DataFile ), 1 ) ) { perror( "Can't _dup2 stdout" ); exit( 1 ); } puts( "This goes to file 'data'\n" ); system( "echo echo echo echo echo echo echo" ); system( "echo echo this goes to 'data' " ); system( "perl -le print(666)" ); // goes to data // Flush stdout stream buffer so it goes to correct file fflush( stdout ); fclose( DataFile ); // Restore original stdout _dup2( old, 1 ); puts( "This goes to stdout also\n" ); puts( "The file 'data' contains:" ); _flushall(); system( "type data" ); } //~ EUREKA http://v8.googlecode.com/svn/trunk/src/platform-win32.cc int fopen_s(FILE** pFile, const char* filename, const char* mode) { *pFile = fopen(filename, mode); return *pFile != NULL ? 0 : 1; }

        as you can see it works

        This goes to stdout first This goes to stdout also The file 'data' contains: echo echo echo echo echo echo echo this goes to 'data' 666 This goes to file 'data'

        If I knew more about gcc and debuggers, I might try to figure out WHERE perl fudges things , but I don't :)

      I tried it - even that doesn't work in this case on windows ...
Re: STDERR going to string
by pvaldes (Chaplain) on Oct 15, 2011 at 14:05 UTC

    Ok, this is the error that you want to catch, two "," at the end of the line. This is not a perl error, is a bibtex error of course, perl should know nothing about this (but Text::BibTeX should).

    TITLE  = {Arranging Arguments},,

    and if I comment all the fuss about STDERR and simply modify this line I got the bibtex error correctly

    my $entry = new Text::BibTeX::Entry $bib_entry or die $!;

    Update: Is obvious after reading the other answers that I had not understood the question at all...

      Update: Is obvious after reading the other answers that I had not understood the question at all...

      It is ok, at least you realized it on your own :)