http://www.perlmonks.org?node_id=1037858


in reply to Compact data classes

Well, what about SQLite?   It’s public-domain(!), used in cell phones all over the planet, and known to be quite efficient.   Maybe you could make that your primary backing-store.   (Perhaps also this would eliminate the need for sorting?)   Because SQLite is used in many resource-constrained situations, it has a lot of options for making efficient use of memory.   (Here, I’m referring to the software itself, not just its Perl implementation ...)

(Quite seriously, “SQLite is another Swiss Army® knife of computer programming” ...)

A homebrew in-memory NRU caching scheme is fairly easy to construct in Perl ... if it turns out that you actually need one when using SQLite.   (Find out, first.)   A hashref can provide random access to strings, while a separate array of hashrefs (to the same Perl objects) provides for round-robin recycling:   when the array has reached its arbitrarily-set limit, shift or pop an element off, delete the key from the hashref (you must do both!), then let the now-unreferenced data disappear into the gloom while you create another key, add a reference to it to the hash, and unshift or push another reference to the same thing onto the array.   (The data-records now have a reference-count of 2, one from the hash, the other from the list, as they will for the duration.)   It’s NRU = Not Recently Used, not LRU = Least Recently Used, but it’s all resident-memory, so, so what.

Replies are listed 'Best First'.
Re^2: Compact data classes
by BrowserUk (Patriarch) on Jun 08, 2013 at 15:46 UTC

    How much memory is used by constructing an SQLite DB with 15,000 records containing 10 x 15-char fields?

    You recommend this module so often; surely you know this off the top of your head?

    Even if not, it will be the work of minutes to try it out won't it?

    Come on. Prove me wrong. Show me that this isn't just another case of a keyword triggering one of your 6 remaining synapses to fire, and cause you to trot out the associated, autonomic "advice".


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      Please limit your discussions on this forum totechnical objections, not vilifying of fellow Monks.   You have been courteously asked this before.   Thank you.

        Please limit your discussions on this forum to technical objections

        Pointing out that your response in no way answers, nor even relates to, the OP question; *is* a technical objection.

        And you've been ask (originally courteously; and only discourteously once it was clear that it was required for you to take the slightest notice), to stop regurgitating pointless boilerplate, in response to questions to which it barely relates, if at all.

        As for "fellow Monks": a) Drop the 's'; b) you're no fellow of mine; nor anyone else who's primary purpose here is to help people.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re^2: Compact data classes
by creeble (Sexton) on Jun 10, 2013 at 18:09 UTC
    And just for the record, while I appreciate that SQLite is free and easy, I'm not looking for a "backing store" per se -- I need this 'db' to be all in ram, and my experience with SQLite using an in-ram database hasn't shown it to be particularly memory-efficient.

    That said, I'd be happy to be proven wrong; I haven't used SQLite from Perl in many years.