Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re^3: RFC: Abusing "virtual" memory

by jbert (Priest)
on Nov 27, 2007 at 17:47 UTC ( [id://653320]=note: print w/replies, xml ) Need Help??


in reply to Re^2: RFC: Abusing "virtual" memory
in thread RFC: Abusing "virtual" memory

Ah, cool, thanks.

In which case the question is more simple: "why sort your keys in the app when you can get the db to do it for you?" :-)

Replies are listed 'Best First'.
Re^4: RFC: Abusing "virtual" memory
by sundialsvc4 (Abbot) on Nov 28, 2007 at 04:46 UTC

    “Why sort your keys externally when you can get the DB to do it for you?”

    Indeed...

    To seek to answer that question, I give you today's workload of 11,344,209 telephone call records. You have exactly 4 hours wall-time to process them. If you imagine that you have enough time to put all those records into a B-tree-indexed random file, I have a bridge to sell you. Instead, to solve this problem and to do so consistently on a daily basis, it will be necessary for you to accomplish the same workload -- with utter reliability and consistency -- much faster.

    It may come as an utter and complete shock to you to fathom that your grandfathers, armed with nothing more punched-card tabulators and sorters, with nary a digital computer in sight, could do that. Every day. Under wartime conditions.

      #!/usr/bin/perl use strict; use warnings; use BerkeleyDB; my $db_file = '/home/snowhare/perl_monks/example.db'; unlink $db_file; my $db = BerkeleyDB::Btree->new( -Filename => $db_file, -Cachesize => 700_000_000, -Flags => DB_CREATE, ); srand; for(my $count = 0; $count < 12_000_000; $count++) { my $random_value = rand(16776216); my $status = $db->db_put( "$random_value" => "$count" ); } undef $db; [snowhare@blue-bay perl_monks]$ time ./big_btree.pl real 3m47.121s user 3m7.978s sys 0m4.884s
      This was on a desktop class machine with 1.5GBytes of RAM and an AMD Athlon 64 3000+ processor running Fedora Core 6 Linux.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://653320]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others exploiting the Monastery: (8)
As of 2024-03-28 09:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found