You want to know my trick for that? When debugging, instead of outputting with warn $message, output messages with ::diag $message.
::diag outputs messages prefixed with a "#" sign, so it looks nice in test output.
::diag won't cause Test::Warnings fails.
When you're not running your test suite, Test::More probably won't be loaded, so the ::diag function won't exist. This will cause ::diag $message to bomb loudly, so you won't forget any debugging messages you've scattered amongst your code.
Note that because of the way function calls are parsed in Perl, including parentheses in the function call like ::diag($message) will make it bomb at run-time instead of compile-time. This is why I recommend not using the parentheses in ::diag $message. If you need to resolve ambiguities using parentheses, you can always do so with external ones like (::diag $message).
Along the same lines, a statement that starts with :: looks weird, so will stand out like a sore thumb when you're looking for it to remove it.