http://www.perlmonks.org?node_id=287850

If I look in perlvar it says:

$^M By default, running out of memory is an untrappable, fatal error. However, if suitably built, Perl can use the contents of $^M as an emergency memory pool after die()ing. Suppose that your Perl were compiled with - DPERL_EMERGENCY_SBRK and used Perl's malloc. Then $^M = 'a' x (1 << 16); would allocate a 64K buffer for use in an emergency. See the INSTALL file in the Perl distribution for information on how to enable this option. To discourage casual use of this advanced feature, there is no English long name for this variable.

Okay, of to look in INSTALL.... Nada! Not a dicky bird!

Soooo....grep the entire source tree.

changes5.004: Title: "Document $^M in perlvar" emacs\cperl-mode.el:$^M A buffer for emergency memory allocation w +hen running out of memory. lib\text\balanced\t\extvar.t:$^M; malloc.c: # Enable code for an emergency memory pool in $^M. See p +erlvar.pod malloc.c: # for a description of $^M. malloc.c: # PERL_EMERGENCY_SBRK is enabled) without touching $^M. +See malloc.c: # perlvar.pod for a description of $^M. malloc.c: PerlIO_puts(PerlIO_stderr(),"Bad alignment of $^M!\n" +); malloc.c: PerlIO_puts(PerlIO_stderr(),"Bad alignment of $^M! +\n"); pod\perl5004delta.pod:=item $^M pod\perl5004delta.pod:compiled for this, Perl may use the contents of +C<$^M> as an emergency pod\perl5004delta.pod: $^M = 'a' x (1<<16); pod\perl5004delta.pod:variable C<$^M>. See L<"$^M">. pod\perl5004delta.pod:However, if compiled for this, Perl may use the +contents of C<$^M> as pod\perldiag.pod:However, if compiled for this, Perl may use the conte +nts of C<$^M> as an pod\perltoc.pod:$SYSTEM_FD_MAX, $^F, $^H, %^H, $INPLACE_EDIT, $^I, $^M +, $OSNAME, $^O, pod\perltoc.pod:$^E, $^H, $^M pod\perlvar.pod:=item $^M pod\perlvar.pod:However, if suitably built, Perl can use the contents +of C<$^M> pod\perlvar.pod: $^M = 'a' x (1 << 16); win32\html\pod\perl5004delta.html:<dt><strong><a name="item__24_5em">$ +^M</a></strong><br /> win32\html\pod\perl5004delta.html:compiled for this, Perl may use the +contents of <a href="#item__24_5em"><pre>$^M</pre></a> as an emergenc +y win32\html\pod\perl5004delta.html: $^M = 'a' x (1&lt;&lt;16);</pre> win32\html\pod\perl5004delta.html:variable <a href="#item__24_5em"><pr +e>$^M</pre></a>. See <a href="#__m">$^M</a>. win32\html\pod\perl5004delta.html:However, if compiled for this, Perl +may use the contents of <a href="#item__24_5em"><pre>$^M</pre></a> as win32\html\pod\perldiag.html:However, if compiled for this, Perl may u +se the contents of <a href="file://e|\perl5.8.1-rc4\html/pod/perlvar. +html#item__24_5em"><pre>$^M</pre></a> as an win32\html\pod\perltoc.html:$SYSTEM_FD_MAX, $^F, $^H, %^H, $INPLACE_ED +IT, $^I, $^M, $OSNAME, $^O, win32\html\pod\perltoc.html:$^E, $^H, $^M win32\html\pod\perlvar.html:<dt><strong><a name="item__24_5em">$^M</a> +</strong><br /> win32\html\pod\perlvar.html:However, if suitably built, Perl can use t +he contents of <a href="#item__24_5em"><pre>$^M</pre></a> win32\html\pod\perlvar.html: $^M = 'a' x (1 &lt;&lt; 16);</pre>

Hmm. Essential several repeitions of the same passage from perlvar... ponder... what other keywords... PERL_EMERGENCY_SBRK. Try that and you get exactly the same hits as for $^M.

Then I thought about it a little.

If I assign $^M = chr(0) x 2**20; what does that do? From the (sparse) documentation it suggests that it grabs a meg of ram and holds it in reserve... then, if at somepoint during my program something I do causes perl to try and allocate some memory and it fails, it has a 1 MB 'reserve' that will allow it to continue....

Hang on!

If I grab 1 MB up front, to give perl a reserve to draw on, doesn't that just mean that malloc will fail 1 MB earlier than it would have?

So great, now perl knows it's running out of memory and has switched to the reserve tank.... but unless I--or rather my program does something different--that just means that it will carry on doing what it's doing and run out anyway! And without some way for my (program) knowing that perl has switched to the reserve tank, I'm gonna carry on tanking down the highway to oblivion.

Hmmm. Maybe there is someway to 'obtain' that notification... I could wrap everything that causes memory allocation into an eval and watch for the infamous "out of memory" error? Gonna make the program kind of messy, not to mention the problems with scoping.

Maybe I could tie $^M and trap attempts to access it? Will such magic be invoked if it is perl that is accessing it? Seems unlikely.

April Fool!

Is there a way to use $^M to good effect, or is the use of the word 'advanced' in

To discourage casual use of this advanced feature, there is no English long name for this variable.

a euphemism for "April Fool!"?


An aside

As an aside, recently, when I needed to test huge file support, I discovered that I simply didn't have enough free space on any given partition on my drive to accomodate a file of > 4 GB, which was a pain. Then inspiration struck. My filesystem supports compression. If a file consists of all the same char, it should compress pretty well, so I used a perl one-liner to write a 10 GB file that consisted of all nulls (chr(0)), and sure enough it worked. I have a file that perl reports the size of as 10 GB but that only consume 5 MB of diskspace. Perfect for testing.

That made me think about my swap file. If I create my swap file on a compressed partition, and then use perl to allocate say 100 huge scalars my $buf1 = chr(0) x 100 * 2**20;, then once each scalar gets swapped out, it would be compressed pretty small.

Now I have a program running with 10 GB of memory, which ain't bad for a 32-bit box. All I need now is a way to make use of all that RAM:)


Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller
If I understand your problem, I can solve it! Of course, the same can be said for you.

Replies are listed 'Best First'.
Re: Is $^M a leftover April Fool?
by perrin (Chancellor) on Aug 30, 2003 at 00:30 UTC
    As I understand it, this feature exists to allow you to try and figure out where and why your program is running out of memory, and maybe do some emergency cleanup. Perl will die() when it fails to malloc, but the die can be caught by a DIE signal handler which can print a stack trace. It's not very fashionable to use DIE handlers these days because of the tricky issues around using them with eval{} blocks, but it seems like you could get it to work if you really needed it.
      The docs seem to support what you said, but according to BrowserUK theres no trace of that functionality in the actual source code. Do you have some counter evidence?

        I wasn't suggesting that it isn't there. As perrin points out, it's right there in malloc.c at the top of the grep listing I included.

        My problem is

        1. The almost complete absence of documentation.
        2. It's useablilty (or lack thereof).

        A little more documentation outlining under what circumstances it can be used--which perl build options are required for it to be enabled etc.--plus some indication of how one might use it would go a long way.

        Perhaps because Win32-perl generally translates to AS perl which doesn't appear to be built with the PERL_EMERGENCY_SBRK option enabled, and the facility hasn't been exercised under Win32.

        Maybe because sbrk(2) is a *nix system thing and Win32 doesn't have any equivalent?

        Try as I might, I can't persuade $^M to do anything, at least not that I can detect.

        It is quite difficult to tell as when you have swapping enabled, you can allocate an aweful lot of memory before you run out, but by the time you do, the system has become quite unresponsive, so actually detecting the point of failure is very hard. I tried setting a %SIG{__DIE__} handler that use Carp::croak() to give me a trace back, but I never saw it.


        Examine what is said, not who speaks.
        "Efficiency is intelligent laziness." -David Dunham
        "When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller
        If I understand your problem, I can solve it! Of course, the same can be said for you.

        It's right there in malloc.c. Look for "emergency_sbrk".
Re: Is $^M a leftover April Fool?
by Elian (Parson) on Sep 01, 2003 at 17:01 UTC
    $^M was an Ilya-thing, which he added in during his work on perl's built-in memory allocation system. Nobody else ever really cared about it so far as I remember from p5p, so it suffers some from both neglect and unfinshedness. I remember in the 5.004/5.005 days that it used to work, but I honestly don't remember how, nor whether it ever worked for anyone other than Ilya.

      Thanks for the info Elian, I'd pretty much come to the conclusion that it was a forgotten avenue, but it's nice to know I'm not just missing the plot somewhere.

      Would raising a bug against the documentation to add a note to the effect that it probably isn't useful outside of the core development team be in order do you think?


      Examine what is said, not who speaks.
      "Efficiency is intelligent laziness." -David Dunham
      "When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller
      If I understand your problem, I can solve it! Of course, the same can be said for you.

        Can't hurt. With the bug in, worst case is it gets ignored. Best case is either $^M is made to work, the docs are updated to reflect how it actually does work (if there's something you missed) or get chopped out.
Re: Is $^M a leftover April Fool?
by rir (Vicar) on Jan 06, 2005 at 14:46 UTC
    $^M By default, running out of memory is an untrappable, fatal error. However, if suitably built, Perl can use the contents of $^M as an emergency memory pool after die()ing.

    This seems a straight forward description. The principle is well known. You've died and want to clean up but you are out of memory and can't allocate space to do what you must. $^M is implemented to address this issue.

    Be well,
    rir

      Here's a blast from the past. You're right, the description is very straight forward. It just doesn't work.

      At least, on the version of perl I was using at the time, and given the near absence of documentation of the 'feature', try as hard as I could, I could see no way of utilising it.

      And, noone ever came forward with any sugggestion of how I can use $^M to allow my Perl program to obtain control once the system refuses the process further memory allocation so that I might at least be able to clean up; close filehandles, put out a "Sorry!" message, prevent the system generating a core dump. You know, that sort of practical step that $^M promises to allow.

      Hmm. What do you call a feature that cannot be used?


      Examine what is said, not who speaks.
      Silence betokens consent.
      Love the truth but pardon error.
        I've not tried it, but I seem to recall it working this way. At the beginning of your program, set $^M to a huge pad string:
        BEGIN { $^M = "x" x 65536 }
        Then, when you're about to take an "out of memory" hit, Perl starts the normal death mechanism, but first deallocates the $^M variable, hopefully permitting all your death handlers to do their job, because there's now 64K of breathing room to allocate temps and stuff.

        At least, this is what I recall.

        -- Randal L. Schwartz, Perl hacker
        Be sure to read my standard disclaimer if this is a reply.

Re: Is $^M a leftover April Fool?
by bluto (Curate) on Jan 06, 2005 at 22:42 UTC
    About 20 years ago when I was programming early Macs, this kind of feature was used since the memory size was small and there was no swap file. I think it was called a "rainy day fund", or something else just as quaint.

    A memory allocation error probably means that most likely (1) you've been leaking memory for a long time and you asked for a little bit more; (2) you passed in a garbage value (something huge) to the allocator; or (3) the heap is corrupt. With today's cavernous virtual memory, 1 is much less common and can usually be detected long before it happens so this feature wouldn't be useful there. 2 is probably pretty rare, but it might help there. With 3, it could even be dangerous to continue (e.g. unflushed open disk files) since you'll be touching the heap at least once and it's already bad.

      I'm not saying that the feature couldn't be useful.

      I am only saying that as it is currently implemented, from my best efforts to understand it, the only time it willl ever be utilised is if in the process of running a perl program, the interpreter attempts to allocate more memory and gets a fail from alloc/malloc/calloc.

      At that point, the perl interpreter may attempt to invoke Perl_croak() in order to report this fatal error.

      As Perl_croak() is a wrapper around sprintf(), it may need to allocate a buffer into which to format the error text, and if the Perl programmer has had the foresight to preallocate a reserve to $^M, then Perl_croak() may be able to allocate that buffer when it might otherwise have failed.

      There are two things wrong with that though.

      • As far as my exploration of the source has gone, the only text that is ever issued when a memory allocation attempt fails, is "Out of memory".

        As far as I am aware, there is never any attempt to relate the error back to a line in the Perl source that instigated the memory allocation that failed. It would probably be near impossible to do that.

        In the absence of any variable information, the need to allocate memory in order to die seems minimal.

      • Even if the message produced at the point memory allocation failed did contain variable information (line numbers, traceback, whatever), it would surely be possible to hardcode a preallocated buffer space of 256 or 512 bytes for this purpose and remove the need for the Perl programmer to do that preallocation?

        Even in an embedded system, that sort of buffer is negligable. Years ago, when I needed a reliable buffer in which to construct fatal error traceback information in a highly, memory constrained environment (a device driver that could never occupy more than one 64k segment), the solution I adopted was to re-use the constant string table for this purpose.

        In a process that occupies at least 1.5 MB of ram minimum, given that this buffer is, by definition, only used just prior to the process terminating, there has to be a suitable sized chunk of memory that can be re-used for this purpose somewhere?

      I'm also saying that given that the Perl runtime will know that it is utilising the $^M reserve--if one has been established--it would be possible for it to inform the Perl script, perhaps via one of the unused signal values, and give it a chance to attempt some form of cleanup and/or reporting, prior to the final act.

      But mostly I am saying, given the current state of the documentation of this feature, I seriously wonder how many people have ever attempted to use it. And of those that did, how many saw some benefit from it?

      So the program managed to issue the "Out of memory" message and self-terminate, rather than segfault and produce a core dump. But did that actually benefit anyone? Did it save the day? Prevent a greater malady? Help the programmer track down the cause?


      Examine what is said, not who speaks.
      Silence betokens consent.
      Love the truth but pardon error.