This might help:
$ perl -le 'eval { "use gobbledygook;" }; print "ok"'
ok
$ perl -le 'eval { use gobbledygook; }; print "ok"'
Can't locate gobbledygook.pm in @INC ...
$ perl -le 'eval "use gobbledygook;"; print "msg = ($@)" if $@; print
+ "ok"'
msg = (Can't locate gobbledygook.pm in @INC ...)
ok
What you were doing was evaluating the code (the braces make it a code block) that contains a non-empty string; the non "0", non-empty string evaluates true, and the eval block doesn't error, so $@ doesn't get set. What you need to do is either evaluate block of real code in braces, or evaluate a string, not evaluate a string inside a block of code. The second and third in my example both try to use the module; the first (equivalent to your code) does not try to load the module.