Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??

Leveraging CountZero’s comment especially here:   “55 million records” definitely need to be indexed to be useful, but an in-memory hash table (might or ...) might not be the best way to do it, if only because it obliges the entire data structure to be “in memory,” with all of the potential (and, rather unpredictable) impact of virtual-memory paging should the system become memory-constrained.   (Also, and mostly for this reason, it does not scale well ...)

Yes, a hash-table, or tables, probably is the best “in-memory” choice.   The design question is ... is “in-memory” the best choice?   I suggest that it might not be.

If you put these data into any sort of database, the records are, first of all, stored on disk, which has unlimited capacity.   And yet, the data are indexed ... also on-disk.   The database can locate the records-of-interest and then bring those into memory for further processing as needed.   If the database grows to 100 or 1,000 times this number of records ... well, it will just take a little longer, but it won’t fail.   You want to design systems that do not degrade severely (and unexpectedly) as volume increases.   “In-memory” approaches are notorious for “hit the wall and go splat” when memory runs short and paging kicks in.   So, you either need to establish confidence that this won’t happen to you in-production at worst-loads, or design in some other way to mitigate that business risk.


In reply to Re: Scaling Hash Limits by sundialsvc4
in thread Scaling Hash Limits by Endless

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others examining the Monastery: (7)
    As of 2014-09-17 21:40 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      How do you remember the number of days in each month?











      Results (99 votes), past polls