I've just run a (not yet thoroughly tested; nor cleaned up; nor even fully optimised) solution to my problem, and the results are promising:
C:\test>BiMap.pl -S=5 -F=0.6 -I=33554432
Fetch by Addr & Fetch by Name took 18.535743 seconds for 11881376 item
+s in a 33554432 sized BiMap
Mem: 965,028 K
For context, here is the single Perl hash solution from the other thread on the same dataset: C:\test>1112165-hash.pl
Start: mem: 7,892 K
23762752
Hash built: mem: 4,382,916 K
Fetch by Name took 7.166012 seconds
Fetch by Addr took 7.895133 seconds
Hash queried: check mem: 4,382,940 K
That equates to just 7% longer to run the same dataset using just over 1/5th the memory. Not bad for a fairly crude POC with still considerable potential for further time reduction.
So, I've expended 3 days -- call it ~50 hrs of my time -- to optimise this.
No matter how you look at it, that is a far better trade-off than your criminally inept suggestion -- to move to a disk-based DB --
which at (a minimum) of 5000x slower, would cost my customer $2800 * 5000 = $14 million for every hour of processing he intends to do.
And given the proposal is to run 100 hours of simulations; the net cost of your suggestion would be $1.4 billion - $280,000 = $1.12 $1.39972 *BILLION* of your customer's money that you would be flushing down the drain; and all because you've eschewed learning anything new for the last 30 years;
seemingly preferring to remain more an archaeologist, than a programmer.
I just hope that people will learn from this and finally realise that your malodorously constructed, maliciously provocative, willfully ignorant, hopelessly outdated -- not just non-useful, but perilously inept -- completely worthless, merit-less fluff; is intended to be neither technically useful nor intellectually stimulating; its only purpose being shameful self-promotion of the vilest, any-hits-are-good-hits, form.
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| 