Does this seem reasonable?
No. It seems very dangerous to me.
XS modules can be binary incompatible between two perls that were compiled with different flags. For example the perl that comes with Ubuntu is compiled with -Dusethreads, and so XS modules compiled for that perl will assume that perl is threaded. Now if the perlbrew-installed perl wasn't configured with threads, and tries to load a threaded XS module, things could go very badly.
In the best case you'll get a segmentation fault pretty quickly. In the first case you'll simply get memory corruption and thus output values that are wrong in subtle ways, and you will have a very hard time finding and diagnosing those errors.
And there are many more flags than just threading, so much potential for things to go wrong.
So, don't make such a fallback. Instead simply install missing modules for the perlbrew installation when needed. Run
and then your modules are just a cpanm Desired::Module away.