I’m not sure what your intended algorithm is. That is to say, it is not readily apparent what set of bright-line rules lead from the input to the output shown.
Nevertheless ... surely there must be some misconception about “references,” that you would say (at the same time) “but that’s not what I want” and “I want to do it efficient.” References are exactly the solution to this sort of problem, where you want (large ...) values to be in several places at one time. (Note: this presupposes that you do not need to modify the values independently.)
We need more detailed information about exactly what you are trying to do here, with more detail than what you have given us here, because there are a great many competing factors to consider in the choice of an appropriate data-structure ... a choice that will be a deal-breaker for this project.