Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re^4: RFC: Abusing "virtual" memory (page faults vs malloc)

by tye (Cardinal)
on Nov 28, 2007 at 02:14 UTC ( #653451=note: print w/ replies, xml ) Need Help??


in reply to Re^3: RFC: Abusing "virtual" memory (page faults vs malloc)
in thread RFC: Abusing "virtual" memory

I won't try to interpret a load of documentation and try to decide how much of it does or doesn't apply to how Perl does things, etc. A trivial test shows your conclusion is wrong:

#!/usr/bin/perl sleep 10; $#a= 100; sleep 10; $#b= 1000; sleep 10; $#c= 10000; sleep 10; $#d= 100000; sleep 10; $#e= 1000000; sleep 10; $#f= 10000000; sleep 10;

Fire up Task Manager then launch the above script. Find "perl.exe" and look at the "Page Faults" column (add it if it isn't there). My results were:

1981 1984 +3 1986 +2 1996 +10 2095 +99 3077 +982 12865 +9788

So pre-allocating to 10-times the size requires 10-times as many page faults. Not 1.

The documentation supports your use of the word "committed" but I stand by my claim that pages are not "committed" such that using them won't incur a page fault. It appears (from my experiment) that even the first use of a freshly "committed" page incurs a "page fault" in order to make the page available. I realized that this initial "page fault" might be different such as between different pools, etc, but I didn't want to get into obscure details that only apply to one platform; so I stayed rather vague on that particular point to be safe. But testing shows that for my Perl on my Win32, the number of page faults for extending an array is propotional to how much you extend it.

So yes. Even accesses to commited memory can cause a pagefault in low memory situations,

No, I was not in a "low memory situation" here, IMHO. My system was relatively idle. I re-ran the scenario with almost nothing else running and got similar results (643 646 647 657 755 1734 11519).

- tye        


Comment on Re^4: RFC: Abusing "virtual" memory (page faults vs malloc)
Select or Download Code
Re^5: RFC: Abusing "virtual" memory (page faults vs malloc)
by BrowserUk (Pope) on Nov 28, 2007 at 10:01 UTC

    This displays the page fault count before and after allocating an 40960 byte chunk of memory from a heap, then freeing it and allocating it again.

    #! perl -slw use strict; use Inline C => Config => LIBS => '-lpsapi.lib'; use Inline C => 'DATA', NAME => 'heap', CLEAN_AFTER_BUILD => 0; my $heap = heapCreate( 0, 0, 1024 * 1024 ); my $space = heapAlloc( $heap, 0, 4096 * 10 ); heapFree( $heap, 0, $space ); $space = heapAlloc( $heap, 0, 4096 * 10 ); print heapSize( $heap, 0, $space ); __DATA__ __C__ #include <windows.h> #include <psapi.h> U32 heapCreate( U32 flags, U32 initial, int max ) { return (U32) HeapCreate( flags, initial, max ); } U32 heapAlloc( U32 hHeap, U32 flags, U32 size ) { U32 pMem; PROCESS_MEMORY_COUNTERS pmc; pmc.cb = sizeof( PROCESS_MEMORY_COUNTERS ); GetProcessMemoryInfo( GetCurrentProcess(), &pmc, sizeof( PROCESS_MEMORY_COUNTERS ) ); printf( "pagefaults before alloc of %d bytes: %d\n", size, pmc.PageFaultCount ); pMem = (U32)HeapAlloc( (HANDLE)hHeap, flags, (SIZE_T)size ); GetProcessMemoryInfo( GetCurrentProcess(), &pmc, sizeof( PROCESS_MEMORY_COUNTERS ) ); printf( "pagefaults after alloc of %d bytes: %d\n", size, pmc.PageFaultCount ); return pMem; } U32 heapSize( U32 hHeap, U32 flags, U32 mem ) { return (U32)HeapSize( (HANDLE)hHeap, flags, (LPVOID)mem ); } U32 heapFree( U32 hHeap, U32 flags, U32 mem ) { return (U32)HeapFree( (HANDLE)hHeap, flags, (LPVOID)mem ); }

    The output

    c:\test>HeapMem.pl pagefaults before alloc of 40960 bytes: 952 pagefaults after alloc of 40960 bytes: 964 pagefaults before alloc of 40960 bytes: 964 pagefaults after alloc of 40960 bytes: 964 40960 c:\test>HeapMem.pl pagefaults before alloc of 40960 bytes: 953 pagefaults after alloc of 40960 bytes: 965 pagefaults before alloc of 40960 bytes: 965 pagefaults after alloc of 40960 bytes: 965 40960 c:\test>HeapMem.pl pagefaults before alloc of 40960 bytes: 953 pagefaults after alloc of 40960 bytes: 965 pagefaults before alloc of 40960 bytes: 965 pagefaults after alloc of 40960 bytes: 965 40960

    Shows that the first time, allocating 10 pages of memory results in 12 page faults. The second time none.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      Duh. That doesn't mean that your program trying to use lots of physical memory is going to do less page faulting. The original problem statement:

      By loading several million records into an array, the virtual-memory footprint of this application blossomed to about 44 megabytes, which caused about 75,000 page faults to occur just in loading and sorting that “memory” list. Although there was enough RAM to allow that to happen without physical I/O, Linux discarded a corresponding amount of file-buffer space ... and Linux depends a great deal on its buffers for good performance.

      So your "solution" won't prevent the problem. Granted, pre-allocating can reduce heap fragmentation and eliminate a few copyings, but it isn't going to make several million records suddenly no longer require many MBs and suddenly leaving room for most of the file buffer to remain. And a fragmented heap is likely to have less impact on the number of pages that need to be kept swapped in than it does on the total heap size. So I stand by my prediction that such games are unlikely to have a major impact in this type of situation.

      - tye        

Re^5: RFC: Abusing "virtual" memory (page faults vs malloc)
by amarquis (Curate) on Nov 30, 2007 at 14:17 UTC

    Side note for monks who, like me, are crap in a Win32 environment: You can add all sorts of different columns to the process list in task manager through the view->select columns menu.

    I didn't know that before this node and it took me a few minutes of right-clicking on things within the pane to figure it out.

      Even better, try Process Explorer (which can replace Task Manager). I added it to a USB flash drive that has utilities to save my sanity when forced to use Windows.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://653451]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (8)
As of 2014-07-11 00:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    When choosing user names for websites, I prefer to use:








    Results (217 votes), past polls