The loaded modules aren't likely consuming much of import. Rather than prematurely optimize, either figure out how to measure the impact or find something actually useful to spend your time on.

I can see value in delaying the loading of such a large number of modules. But I find it hard to imagine how trying to unload them again would do anything other than consume more resources. The pages allocated to your process that your process doesn't touch are one of the least important resources in most situations are so are almost never worth trying to exert effort to optimize them.

For going from 138 modules loaded down to 4, it may even be more efficient to use "exec( $^X, $0, @ARGV );" to do that instead. And that is also less likely to cause breakage when you try to re-load the modules. Though, I wouldn't even bother with that unless I could find evidence of some real problem.