Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^5: write hash to disk after memory limit

by sundialsvc4 (Abbot)
on Mar 16, 2015 at 23:16 UTC ( #1120233=note: print w/replies, xml ) Need Help??


in reply to Re^4: write hash to disk after memory limit
in thread write hash to disk after memory limit

I definitely want to suffix this post with all of the extremely-valid points that BrowserUK made in his comment concerning “all the implications of ‘adding an extra stick.’”   As his Voice Of Experience™ says, there are a lot of compelling reasons why my “point #1” might not apply to “your case, whatever it is.”

My essential point ... which was intended to be more generic ... is basically this:   “either you must buy the hardware necessary to make [most of ...] the data really be “in memory,” or you must redesign your approach to demand less RAM.   The one thing that you cannot do, IMHO, is to remain where you are.   The operational characteristics of both a hash-table and Perl’s memory-manager work cripplingly against you if physical memory is actually “grossly overcommitted,” as is the case here.

Effectively, the storage-pool that is being accessed by your application is “disk” ... not “memory.”   And the operating system’s virtual memory manager is being asked to “graciously do” ... the impossible.   It simply isn’t credible to pretend that the memory resource is “actually ‘real,’” when it is more than 200% over-committed.

The application’s fundamental design, unfortunately, is flawed.   However, if you know that the problem-to-be-solved never will “grow ten-fold,” it certainly will be fast!   Hence, my suggestion to seriously explore “throwing silicon at it.”   (Even $10,000 worth of hardware is less expensive than $10,000 worth of developer-time.)   However, if the problem might grow, you are ... ahh ... scr00d.™   You’re going to have to re-design this thing, maybe from the ground up.

  • Comment on Re^5: write hash to disk after memory limit

Replies are listed 'Best First'.
Re^6: write hash to disk after memory limit
by BrowserUk (Pope) on Mar 16, 2015 at 23:52 UTC
    (Even $10,000 worth of hardware is less expensive than $10,000 worth of developer-time.)

    Amongst all the other fallacies in your posts in this thread, that one deserves requires refutation.

    $10,000 == $10,000. QED. B***shit!

    Further, $10,000 of hardware may fix the problem for one user; but it doesn't help the user in the next cubicle over; or the one down the corridor; or the 1001 scattered to the 4 corners.

    However, $10,000 of developer time could fix the problem for all of them.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
    In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
      It took three attempts at invoking you (1, 2, 3), I guess he finally succeeded. Too bad, I was hoping you had given up on feeding him. You're the highest-ranking monk on this site after vroom; he clearly enjoys having your attention.
        It took three attempts at invoking you (1, 2, 3), I guess he finally succeeded.

        Yeah. D'ya think he wants to be my bitch?

        Too bad, I was hoping you had given up on feeding him.

        But ya missed the point. My post was all about "invoking" you!


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
        In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1120233]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (7)
As of 2019-11-18 08:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Strict and warnings: which comes first?



    Results (88 votes). Check out past polls.

    Notices?