Measure. Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest. ~Rob Pike

What has measurement shown the difference is between conditionally loading the modules and just loading them all to begin with? If it's not worth measuring, it's not worth optimizing. You then have to weigh that difference in performance against making your code less clear, less maintainable, more complex and less testable.

I think you'll find that the most straightforward strategy of loading all your modules at the beginning of the file makes the most sense.

A couple additional notes:

If your module loads a large number of only loosely-related modules, a code reorganization may be a better strategy. But even that may be more trouble than it's worth unless there are additional reasons to do it.

If your module is large with lots of subroutines, you might consider adding a comment to each subroutine about what module(s) it uses. That makes it easy to detect if there are any loaded modules that are no longer used in your subroutines.