Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
From your previous question I got:

Each object entry requires at least one FETCH (to see if the object already exists), and one STORE...

My question is, Is this necessary? I don't know what you are trying do accomplish, but if memory is not an issue, and keeping up with SQUID is, then you want as fast a process as possible writing to your DB as SQUID runs. I haven't noticed if you've said anything about the time constraints on actually using this data... who or what is querying the datastructure after you build it? do they care if it takes longer to query? do they care if the order of complexity for a query is increased?

if not, then we can fix you in no time... just don't do the FETCH op. STORE everything, and when you go to read the data (A) query all the lines you need, (B) delete them, (C)build your tree and give it to the user, and (D) re-insert the tree while the user is playing with it.

also, you potentially could change the datastructure... if you can generate an md5 in constant time, and memory is not a problem, you could generate an array of the length of possible checksums, then you could read and write to it in constant time. more realisticly if memory is not an issue you could play with very large arrays, indexed on the first N chars of the checksum...

point is, instead of building something that scales up well, you could build something that doesn't scale at all, and is just always going to be bigger than your dataset. that'll always win for speed.

In reply to Re: Performance quandary by Xanatax
in thread Performance quandary by SwellJoe

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others chilling in the Monastery: (5)
    As of 2014-11-21 04:41 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      My preferred Perl binaries come from:














      Results (105 votes), past polls