"Obv reasons"? I'm not sure I follow. In a day and age where Java seems all too popular (I currently have one Java app running that is using 1116m virtual and 657m resident), I don't exactly follow why ~80-100m in memory should be a concern.
Especially since the guys who wrote your OS, whether that's Windows, Linux, or BSD (Mac), or pretty much any other modern OS, have already solve the problem of using a hard disk as if it were RAM. So if you really do run out of memory and start swapping, it actually can often be faster than if you try to be sneaky. Usually, the OS will swap out some other process first while yours runs, which means that you'll get to stay all in memory.
I suppose my suggestion is to start with what works, and worry about the optimisations later. You may not really actually need them. Do it all in memory since that's probably way simpler. Optimise it later.