$? set to strange values on failure under Win32

[pjf]

Wise monks,

I've been spending my time recently working upon a new revision of IPC::System::Simple, a module designed to take the headache out of calling external commands. Unfortunately, I've hit a snag under Windows.

My Windows system (WinXP, ActivePerl 5.8.8) appears to assign a value of 1 << 8 to $? upon failure to start a process. The following example demonstrates this:

system("this_command_does_not_exist",1);
printf("raw: %d ; apparent exit: %d\n",$?, $?>>8);

Unfortunately, this is indistinguishable from the command running to completion and returning an exit status of 1. Not being able to tell the difference is a very bad thing. There's also a troublesome error message that is printed ('...' is not recognised as an internal or external command) that I assume is from the Windows shell, and ideally I'd love to supress the printing of this message.

Using the multiple argument version of system does not alter these results.

I'm sure there must be a way under Windows to tell if a command did not start, and it's probably something simple, but I'm at a loss to find it.

Any tips and pointers would be greatly appreciated.

Update: This problem isn't exactly unique to Windows. When using single-argument system() under Unix with shell meta-characters, the result in $? reflects the exit value of the shell, which returns 127 on a failure to find the command. This is indistinguishable from a successfully executed command returning an exitval of 127.

[BrowserUk]

You can distinguish between a successfully run command returning 1 and a failed command by checking $^E.

If $^E is empty, the command ran successfully and the contents of $? are the lower 8 bits << 8 of the 16-bit value returned by the executable.

If $^E is set, usually to The system cannot find the file specified, then the command could not be found--either directly, nor by running cmd.exe passing the parameter(s) supplied on the system call as argument(s) in the hope it can be resolved via the path--and $? contains the failure code (1) from cmd.exe.

This is when the ...' is not recognised as an internal or external command message is produced. This can be suppressed by redirecting stderr to null as in the 3rd example below:

#! perl -slw
use strict;

print 'Successful command returning 1';
system 'c:/perl/bin/perl.exe -e"sleep 5; exit 1"';
print "![$!] ?[@{[$?, ':', $?>>8]}] E[$^E]";

print "\nNonexistant command, attempted direct but fallback to via cmd
+";
system 'c:/doesNotExists.exe';
print "![$!] ?[@{[$?, ':', $?>>8]}] E[$^E]";

print "\nNonexistant via cmd because of the presence of shell chars se
+nding stderr >null";
system 'c:/doesNotExists.exe 2>null';
print "![$!] ?[@{[$?, ':', $?>>8]}] E[$^E]";

__END__
c:\test>junk
Successful command returning 1
![Bad file descriptor] ?[256 : 1] E[]

Nonexistent command, attempted direct but fallback to via cmd
'c:/doesNotExists.exe' is not recognized as an internal or external co
+mmand,
operable program or batch file.
![No such file or directory] ?[256 : 1] E[The system cannot find the f
+ile specified]

Nonexistent via cmd because of the presence of shell chars sending std
+err >null
![No such file or directory] ?[256 : 1] E[The system cannot find the f
+ile specified]

The real problem here is that the win32 implementation of the posix system call attempts to emulate that call as best it can.

So, for example, return codes from executables above 255 get truncated to 8-bits and shifted left to leave room for signal values, which you can never receive.

And even if you supply multiple arguments to system, if the first attempt to run the command directly fails, it defaults to trying again, by supplying the commands you supply, to 'cmd.exe' to see if it can find the command.

This is not an exhaustive explanation of what goes on in Win32.c. I seem to recall tye posting a pretty comprehensive breakdown, but I can't persuade google to find it for me.

You might also want to look into using SetErrorMode() to (optionally) suppress system popups for things like segfaults and 'Insert disk for drive x:' and similar..

[syphilis]

If $^E is empty, the command ran successfully

Yes, but the command can run successfully and, yet, there will still be an error message in $^E (which is merely the last error, n'est-ce pas ?). When I run the script you provided I get the following output:

C:\_32\pscrpt>perl try.pl
Successful command returning 1
![Bad file descriptor] ?[256 : 1] E[The specified image file did not c
+ontain a r
esource section]

Nonexistant command, attempted direct but fallback to via cmd
'c:/doesNotExists.exe' is not recognized as an internal or external co
+mmand,
operable program or batch file.
![No such file or directory] ?[256 : 1] E[There are no more files]

Nonexistant via cmd because of the presence of shell chars sending std
+err >null
![No such file or directory] ?[256 : 1] E[There are no more files]

Admittedly 1) and 3) are different, but I wouldn't like to have to work out which had succeeded and which had failed based solely upon the ouptut :-)

I'm at a loss for the moment to understand why the expected error message never appears in $^E for me. (I'm on Windows Vista - but I don't think that's an issue. I'll switch to XP and see what happens.)

Cheers,
Rob
Update:On Windows 2000 I find that BrowserUk's script runs as he has reported. However, if one inserts a system'c:/doesNotExist.exe'; immediately below the use strict; in BrowserUk's script then $^E is no longer useful for determining whether the other three ensuing commands ran successfully or not. (My XP laptop has shit itself again, so I had to resort to Windows 2000).

[BrowserUk]

What results do you get on vista from this version?

#! perl -slw
use strict;

undef $^E;
undef $!;
print 'Successful command returning 1';
system 'c:/perl/bin/perl.exe -e"sleep 5; exit 1"';
print "![$!] ?[@{[$?, ':', $?>>8]}] E[$^E]";

undef $^E;
undef $!;
print "\nNonexistent command, attempted direct but fallback to via cmd
+";
system 'c:/doesNotExists.exe';
print "![$!] ?[@{[$?, ':', $?>>8]}] E[$^E]";

undef $^E;
undef $!;
print "\nNonexistent via cmd because 'shell chars'; sending stderr >nu
+ll";
system 'c:/doesNotExists.exe 2>null';
print "![$!] ?[@{[$?, ':', $?>>8]}] E[$^E]";

__END__
C:\test>junk
Use of uninitialized value in undef operator at C:\test\junk.pl line 4
+.
Successful command returning 1
![] ?[256 : 1] E[]
Use of uninitialized value in undef operator at C:\test\junk.pl line 1
+0.

Nonexistent command, attempted direct but fallback to via cmd
'c:/doesNotExists.exe' is not recognized as an internal or external co
+mmand,
operable program or batch file.
![No such file or directory] ?[256 : 1] E[The system cannot find the f
+ile specified]
Use of uninitialized value in undef operator at C:\test\junk.pl line 1
+6.

Nonexistent via cmd because 'shell chars'; sending stderr >null
![] ?[256 : 1] E[The system cannot find the file specified]

Aside: That Use of uninitialized value in undef operator ... wins the

BrowserUk Award with Special Commendation" in the category of "Most Useless Warning of the Week Month Year Ever".

And that's in a highly contested category. :)

[syphilis]

Hi pjf,
That's a good question, and I'll be interested to see what the Win32 experts can come up with.

As regards the suppression of the error message, one could just redirect stderr to the null device (which you do with 2>nul on Windows) ... but you probably knew about that, anyway.

Other than that, all I can think of is to use Win32::Process instead of system(). If the "system" call launched by Win32::Process::Create() fails (in which case it returns 0) you can access good diagnostics via Win32::FormatMessage((Win32::GetLastError()). And if it succeeds, then you can get the exitcode of the process that was launched with $ProcessObj->GetExitCode($exitcode).

Cheers,
Rob

[garu]

Well, I'm no win32 expert, but I might as well give my $0,02c to the issue, hoping it will improve your research ;-)

First of all, you assumed right as the ('...' is not recognised as an internal or external command) message IS indeed from the Windows shell (I can tell as my Windows is in portuguese and this is the only portuguese message I get when running the example files you guys provided), but you already knew that as 2>NUL would probably not have worked otherwise, I guess ;-).

Now, from a programmer's point of view, I'm sure you know that return values are only set for convention purposes, and you can create a program returning whatever you want upon success or failure. As BrowserUk mentioned, MS-Windows has its own method of error handling. However, apart from some special WINAPI functions like GetLastError, it's not as foolproof as you'd expect. In fact, Microsoft's very own official Command Shell Overview states that:

"If a command completes an operation successfully, it returns an exit code of zero (0) or no exit code."

And I guess people should just comply to this behavior if they want things to work as expected.

I tested the implications of this via the && shell command (that, again acording to Microsoft, runs the command following "&&" only if the command preceding the symbol is successful) and, not surprinsingly, here's what I got:

C:\>perl -e "exit 0" && echo "ok"
"ok"

C:\>perl -e "exit 1" && echo "ok"

C:\>perl -e "exit -1" && echo "ok"

C:\>

As you can see, there was no error whatsoever, and still the cmd.exe shell only thought of it as ok when it returned 0. In fact, the %ERRORLEVEL% system environment variable is supposed to hold the error code of the most recently used command, a non zero value usually indicating an error (from MS's description in the above link), but it simply holds the value returned by the program (in my example it was 0, 1 and -1 respectively).

This is supposed to be well-known by every Windows programmer (is it?), and WinError.h has a comprehensive list of 16000 System Error Codes with their respective descriptions. Signal Handling is in itself another problem: although Microsoft's definition states that all signal-terminated programs should have the return value of 3 (oddly enough, not the correct WinError.h definition - but maybe I messed up on their file versions, or maybe there's a different message table for this, or maybe I just got it all plain wrong), perl returns the actual signal value. For instance:

C:\> perl -e "while (1) { sleep 1; }"
 (now I pressed ^C)
Terminating on signal SIGINT(2)

C:\> echo %ERRORLEVEL%
2

By the way, this is what I got when I ran BrowserUk's code on my WinXP/ActivePerl 5.8.8 (build 820):

C:\>perl pjf-win32.pl
Successful command returning 1
![] ?[256 : 1] E[]

Nonexistent command, attempted direct but fallback to via cmd
'c:/doesNotExists.exe' is not recognized as an internal or external co
+mmand, operable program or batch file.
![No such file or directory] ?[256 : 1] E[]

Nonexistent via cmd because 'shell chars'; sending stderr >null
![] ?[256 : 1] E[]

Just to be sure it was not a "Windows perl implementation issue", I tested it on Vanilla-Perl 5.8.8, getting the exact same messages.

Although I think you'll hit the same deadend in Win32 and Win32::Process, I hope you succeed and let us know how you did it!

Hope I understood your problem correctly, and hope it helped somehow :-)