C:\Users\peter.jones\Downloads\TempData\perl>perl -Ilib -e"print qq(\xe4)"
Σ
C:\Users\peter.jones\Downloads\TempData\perl>perl -Ilib -MDOS::Try -e"print qq(\xe4)"
ä

vs

C:\Users\peter.jones\Downloads\TempData\perl>perl -e"print qq(\xe4)"
Σ
C:\Users\peter.jones\Downloads\TempData\perl>perl -e"binmode STDOUT, ':encoding(Cp437)'; print qq(\xe4)"
ä

And the reason your example doesn't work in your test is the same reason that you wrote the module: you need to have the right encoding on the output of your test script as well as the code of the `...`.

#!perl

use strict;
use warnings;
my $result = `perl -Ilib -MDOS::Try -e"print qq(\xe4)"`;
print "first test: $result\n";

use lib 'lib';
require DOS::Try;
print "second test: $result\n";
__END__
first test: Σ
second test: ä

You can see more if you hex dump the bytes being output from the two variants of the oneliner:

C:\Users\peter.jones\Downloads\TempData\perl>perl -e"print qq(\xe4)" | perl -e "print unpack 'H*',$_ for <>"
e4
C:\Users\peter.jones\Downloads\TempData\perl>perl -e"binmode STDOUT, ':encoding(Cp437)'; print qq(\xe4)" | perl -e "print unpack 'H*',$_ for <>"
84