Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: Perl solution for storage of large number of small files

by diotalevi (Canon)
on Apr 30, 2007 at 14:08 UTC ( #612776=note: print w/replies, xml ) Need Help??


in reply to Perl solution for storage of large number of small files

It sounds like you didn't read the BerkeleyDB documentation. If your record is larger than your page size then the record is stored as an overflow record. I don't know how overflow records are accessed. In your case, perhaps you should try using a 64K page size and trying your benchmark again. The default size is 8K which far below your typical record size.

First, the page size implicitly sets the size of an overflow record. Overflow records are key or data items that are too large to fit on a normal database page because of their size, and are therefore stored in overflow pages. Overflow pages are pages that exist outside of the normal database structure. For this reason, there is often a significant performance penalty associated with retrieving or modifying overflow records. Selecting a page size that is too small, and which forces the creation of large numbers of overflow pages, can seriously impact the performance of an application.

Berkeley DB Reference Guide: Selecting a page size

⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

  • Comment on Re: Perl solution for storage of large number of small files

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://612776]
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (8)
As of 2018-07-19 15:46 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    It has been suggested to rename Perl 6 in order to boost its marketing potential. Which name would you prefer?















    Results (411 votes). Check out past polls.

    Notices?