The size of the module isn't nearly as important as what it does.
For example, one could do incredibly complex calculations in just a few lines of code and have that take several seconds to execute.
On the other hand, in several thousand lines one could also write a blog that only does some simple text processing and manipulation, which would be very fast.
Of course, the modules you load can have an impact. POSIX is notorious for being very large. Loading this would probably lead to an increased memory usage.
The best thing to do is to try it. There are dozens of articles online about stress-testing one's website.
Then, refactor as you said, and try it again. Fix whatever, if anything, is slow. Then try it again.
~Thomas~
"Excuse me for butting in, but I'm interrupt-driven..."