In the earliest days of digital computing, memory was the most-scarce resource.   (After all, magnetic doughnuts could be made only so small, and they had to be strung upon three tiny wires by hand.)   Thus, when external storage devices – disk and tape and drum – were developed, they were referred to as “mass” storage.   Most of us remember the bad old days, of MS-DOS and even earlier, when programs had to be separated into “overlays” in order to be made to fit into the one-and-only place from which instructions could be executed.

Well, fast-forward a few decades and “chips are cheap” now.   Gigabytes if not terabytes of semiconductor storage can fit into a space the size of a pencil eraser.   CPU architectures are especially designed to allow gigabytes of storage to be directly addressed.   Developers are eagerly taking advantage of this, because the days of “overlays” and then “virtual memory” (i.e. “paging”) appear to be long-ago and far-away.   After all, RAM has the unique advantage of being instantaneous.   If the data is available in RAM, disk-I/O does not occur.   (And, if the “disk” storage device is made of semiconductors, at least the “seek latency” and “rotational latency” does not happen, even though most semiconductor devices do have a certain form of “latency” of their own.)

There is a fly in that ointment, however.   RAM capacity is not yet so enormous that concerns about virtual memory can be dismissed outright, especially in production situations where many possibly memory-hungry applications are running on the same box at the same time.   Virtual memory is still with us, and therefore we must be mindful of how to work “with” it and not “against” it, just as we were very-obliged to do in the 1970’s.

When virtual memory is being used, “in-memory” data structures might involve disk I/O.   As you know, physical RAM is divided into equal-sized chunks called “pages,” and each page might be “resident” in memory or it might be “swapped out.”   When any virtual address is touched, a “page fault” might occur, and if so the process will be delayed until the necessary disk-I/O has been completed.   (And, in order to make room for the page, another page might have to be “stolen” from someone and written out to disk … thus, two or more disk-I/O’s must take place before the faulting process is allowed to proceed.)

Virtual memory’s success relies upon the assumption that, while page-faults will occur, they will not occur so frequently that the disk-I/O delays add up too much in practice.   The term is “locality of reference,” and it means that programs typically make memory-references in very concentrated groups.   Once a page-fault is satisfied, things should settle-down for a while as the processes continue to refer, again and again, to the page(s) that have most recently been faulted-in.   “Least Recently Used (LRU)” pages, by comparison, are presumed to be good candidates for page-stealing.   The total set of pages that any process requires in order to run without delay, at any instant in time, is referred to as its current “working set” at that instant.

There is, unfortunately, one data-structure mechanism in particular that flies in the face of “locality of reference,” and therefore of “small and tidy and predictable working-sets.”   That mechanism is:   the hash table.   Perl’s “hashref.”

Hash tables work by permuting a key across some smaller key-space in order to arrive at a single “bucket” that is searched for the target value.   Hash functions are designed to spread the key values more or less uniformly, but randomly, across the key space.   Thus, the hash structure itself can represent a large working-set (although hash algorithm designers, including Perl’s, do seek to constrain this).   But in any case, the hash buckets also refer, by reference, to outside blocks of memory that are obtained using memory allocation functions e.g. “malloc().”   The memory addresses pointed-to by the (already, large) hash table will, over time, become quite-widely distributed.   And so we have a “very random-access” data structure:   a large hash-table referencing an even larger set of memory blocks whose individual addresses are not predictable.   (A highly volatile very-active data structure becomes less and less predictable as the hours and days go by.   Working-set sizes increase quickly.)

(Perl’s designers know their stuff.   They know about these issues and carefully designed an industrial-strength system for all of us to enjoy.   We are well cared-for ... but the issues are still there, and, by definition, always will be.)

Working-set sizes become very large, then.   So, what actually happens when such an application enters service in a production machine that’s using virtual memory?   Unfortunately, it becomes a million-pound elephant … using, shall we say, far more than its fair share of RAM.   A disproportionately large amount relative to the others.   And therefore, both a source of virtual-memory pressure and(!) an especially vulnerable victim of it.   If such a program is to run efficiently (as it was specifically designed to do), it must have “all that RAM.”   But, if it gets what it wants (and must have), the other processes can’t get (and keep) theirs.   Paging activity begins to increase, as does the number of processes that are stuck in page-wait and the frequency that each process is stuck in page-wait.   At a certain point, the processing grinds to a halt.   It “hits the wall.”   It is “thrashing.”   The offending application is especially taking it in the shorts ... being especially big and especially vulnerable, it is “punch-drunk.”   But it’s not the only one.   (If there were any subtle timing-related bugs in this or any other application, this is the time when those kinds of problems will really start to show up.)

Given that, in a virtual memory setting, any “memory” reference can result in “disk” I/O, “memory” must in fact be treated as a “disk.”   Each memory-access, especially any access that might be widely-dispersed from other recent ones, must be considered as possibly taking several milliseconds to occur; not the microseconds or nanoseconds that are usually bantered-about by programmers who like to use the “time” command and discuss the third or fourth digit to the right of the decimal point.

Software developers usually don’t experience these things personally when designing their software:   their systems are the biggest, fastest, and especially the fattest of all.   They’ve got two or three large monitors.   Multiple processors.   Gigs of RAM.   As much as the motherboard will support.   In short, a situation altogether unlike the rack mounted boxes where their brainchildren will labor out their appointed business duties.

To run well, and to do so round-the-clock for days and weeks on end, all applications must be good virtual-memory citizens.   Whether their virtual memory allocations be large or small, their working-set sizes must be small … by design.   There are many possible ways to do that:   storing multiple entries in a single large structure rather than in many small ones; “batching” requests for even in-memory data stores; and, using disk-I/O directly instead of implicitly (as virtual-memory actually does).   All operating systems buffer disk-I/O operations, filling all available RAM with buffers but managing those buffers differently than they do VM page-frames.

Probably the very worst thing that can happen to your program’s design is for it to be very splendid on your machine, but unworkable in production … or even, “a pain in the asterisk in production.”   This requires thinking of RAM as being “a thing that is not without-delay,” from the very earliest stages of your designs.

HTH …

Replies are listed 'Best First'.
Re: RAM: It isn't free . . .
by SuicideJunkie (Vicar) on Jan 07, 2015 at 20:47 UTC

    I have a simple personal solution.

    I develop code on my netbook. If it works fast enough for me there, then people with real PCs and gobs of ram will have no trouble at all :).

      A friend of mine works for a company that does a variation on that. They provide developers with a second PC - a very cheap one, slow and very little (512 MB) RAM - for testing the applications the developers create.

      (The developers resisted, at first, but after the first one cut her deadline-is-next-week crunch time in half, the others started using their test PCs.)

        A reply falls below the community's threshold of quality. You may see it by logging in.
Re: RAM: It isn't free . . . (No, but its damn cheap!)
by BrowserUk (Pope) on Jan 07, 2015 at 21:54 UTC

    You really are incapable of taking a hint! I've ignored your posts on this subject because they are irrelevant to my problem!

    This is a research project; nothing to do with "production".

    It will never "run out of ram" or move into swapping because the size of the -- runtime generated -- dataset is controlled by a command line parameter; which will be set specifically to prevent that.

    It will only ever run on research hardware. Currently my machine during development; but once optimised, on a whole bunch of purchased cloud instances specially configured for research purposes!

    Probably, though it depends on which cloud provider is offering the best terms at the time, AWS EC2 R3.8xlarge instances.

    A few noteable highlights: "r3.8xlarge 32 cores 244 GiB ram 2 x 320 SSD".

    And:

    Use Cases: We recommend memory-optimized instances for high performance databases, distributed memory caches, in-memory analytics, genome assembly and analysis, larger deployments of SAP, Microsoft SharePoint, and other enterprise applications.

    And no amount of you banging on about ancient history, extant wisdoms nor inapplicable generic truths changes the fact that I know my problem space and its requirements; and you know nothing.

    (About this or anything contemporary or useful!)


    (After all, magnetic doughnuts could be made only so small, and they had to be strung upon three tiny wires by hand.)

    And that really explains the Sundial in your user handle and company name:

    Sundial; as-in antiquated technology that was grossly inaccurate, and only came close to being correct about twice a year

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      A reply falls below the community's threshold of quality. You may see it by logging in.
Re: RAM: It isn't free . . .
by Eily (Monsignor) on Jan 07, 2015 at 21:07 UTC

    Since your post probably came from reading BrowserUk's question about Bidirectional lookup algorithm, let's put your theory to the test with the method he used:

    use Devel::Size qw/total_size/; $h{$_} = 1 for "aaaaa".."zzzzz"; $elements_size += total_size $_ for %h; $hash_size = total_size \%h; print <<INFO; Total size of the contained elements: $elements_size Total size of the hash: $hash_size INFO __DATA__ Total size of the contained elements: 1223781728 Total size of the hash: 1167897536
    It sure looks like I did something wrong. But since keys are constant strings (of fixed size), they probably take less space than a string in a scalar (scalar have all sorts of additional information, and a bit of extra space to avoid constant reallocation), so in the end I don't think I made any mistakes here. And I'm saying "probably" because I didn't check in the guts.

    A lot of data in a hash sure takes a lot of memory, same goes for arrays because a lot of data will take a lot of memory unless it's properties allow for a very effective compression. If you only use pure perl, trying to avoid a hash because they are "bad" has very high probability of just being worse (hashes are an excellent compromise between algorythm complexity and memory consumption).

      A reply falls below the community's threshold of quality. You may see it by logging in.
Re: RAM: It isn't free . . .
by mr_mischief (Monsignor) on Jan 08, 2015 at 15:19 UTC

    These days it's very often the case that the machine itself may be virtual and is running only one major application. Everything else will be maintenance and monitoring tasks, and the database, the API, the customer front end, the reverse proxy for the front end, the administration interface, and any mail or chat servers will all be on separate (virtual or actual) servers.

Re: RAM: It isn't free . . .
by Anonymous Monk on Jan 07, 2015 at 21:45 UTC

    Disabling overcommits can be as easy as "sysctl -w vm.overcommit_memory=2". Let's not forget about mlockall either. IOW, where there's a basic problem, usually a solution is also thought of.

Re: RAM: It isn't free . . .
by KurtSchwind (Chaplain) on Feb 25, 2015 at 13:34 UTC

    Why is the Meditation itself getting down-votes?

    I mean, I can see that there is some issue between the original poster and some other users, but I don't see any real controversy over the original meditation.

    --
    “For the Present is the point at which time touches eternity.” - CS Lewis

      Here’s an exercise that might help: Sum up a salient point from that meditation.

        Perhaps Priests should be allowed to have sex

      Because its like being lectured on sex by a celibate priest.

    A reply falls below the community's threshold of quality. You may see it by logging in.
    A reply falls below the community's threshold of quality. You may see it by logging in.
    A reply falls below the community's threshold of quality. You may see it by logging in.