http://www.perlmonks.org?node_id=377093


in reply to Combining Ultra-Dynamic Files to Avoid Clustering (Ideas?)

I would just use BerkeleyDB. It solves this exact problem and can handle terabyte-size databases.