in reply to Re^5: Best way to store/access large dataset?
in thread Best way to store/access large dataset?
So when you say "pull calculations", are you talking about performing calculations in the script or pulling data from the database? If you're doing several million/billion calculations against a datasource, it's probably better to try to do some map reducing in a parallel fashion using something like dynamodb and spark/emr. Pulling the rows won't be so hard but having the database crunch a bunch of numbers gets hairy if it's not optimized as such.
Three thousand years of beautiful tradition, from Moses to Sandy Koufax, you're god damn right I'm living in the fucking past
|Replies are listed 'Best First'.|
Re^7: Best way to store/access large dataset?
by Speed_Freak (Sexton) on Jun 26, 2018 at 22:11 UTC