Re^3: System out of memory while storing huge amount of data

[BrowserUk]

I attribute the difference in MB numbers to greater size of pointers on your 64 bit machine,

Indeed. 64-bit pointers cost heavily.

Especially as, (today and for the immediate future), on any machine less than something like $250k, the top 24-bits and usually more will be zeros. Even worse when you consider that the bottom 4-bits will also be 0.

In an XS module I'm writing, I'm experimenting with storing 64-bit pointers in 32-bit fields. By right-shifting 4 places, I get 32-bit values than can cover 64GB, which is as much memory as I'm going to have in the next say 10 years or so.