What you are thinking of is commonly called a cache. :-)
Caches make (a lot of) sense, whenever the data has temporal and/or spatial locality. Which is the typical case.
Now the thing is, your computer already employs caching at multiple levels. An additional caching scheme may still bring benefits if it makes your working set more compact. But there may be opportunities to arrange your large data in a way that improves locality. It depends on the problem. Also, many algorithms have their "cache oblivious" versions.