Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re: What could cause excessive page faults?

by GrandFather (Sage)
on Mar 11, 2010 at 08:26 UTC ( #827978=note: print w/replies, xml ) Need Help??


in reply to What could cause excessive page faults?

From outside the box it looks like Perl is creating an intermediate list between the two maps and somewhere before 5e5 elements the system starts swapping. Before the swapping starts the page faults happen when more memory is needed to extend the lists giving about 3500 bytes per fault - close enough to a 4K page size perhaps. At some point the lists get too big to stay resident and the system starts swapping with the resultant increase in page faults (now against non-resident pages) and consequent increase in time (due to the page fetches from disk).

What seems very odd is that it is happening on a 64 bit system with (I presume) plenty of ram! However I get very similar results with a 32 bit Windows system and 5.10.1 btw.


True laziness is hard work
  • Comment on Re: What could cause excessive page faults?

Replies are listed 'Best First'.
Re^2: What could cause excessive page faults?
by BrowserUk (Pope) on Mar 11, 2010 at 09:03 UTC

    There is no swapping involved. I have 4GB of ram and well over 2 of that was available at the point the scripts achieved maximum usage.

    Page faults can also occur due to Windows 2-stage virtual memory allocation schema. Virtual memory can be 'reserved' by a process, without being 'commited'. Reservation means that space is reserved within the process address space and page tables within the OS internal structures, but no actual physical memory is yet assigned to back that reservation up. When access is first attempted to a 'reserved' page of virtual memory, a page fault occurs. and the page (or many) must be 'commited' before the memory access completes.

    The way this normally works is, in the executable header, there are 2 values that are used to reserve stack and heap spaces for the process when it it loaded. There are 2 other values which define how much of the reservation gets committed each time a reserved page pagefault occurs:

    C:\test>dumpbin /headers \perl64\bin\perl.exe Microsoft (R) COFF/PE Dumper Version 9.00.21022.08 Copyright (C) Microsoft Corporation. All rights reserved. Dump of file \perl64\bin\perl.exe PE signature found File Type: EXECUTABLE IMAGE FILE HEADER VALUES 8664 machine (x64) 5 number of sections 4B60BA96 time date stamp Wed Jan 27 22:13:42 2010 0 file pointer to symbol table 0 number of symbols F0 size of optional header 23 characteristics Relocations stripped Executable Application can handle large (>2GB) addresses OPTIONAL HEADER VALUES ... 1000000 size of stack reserve 100000 size of stack commit 1000000 size of heap reserve 100000 size of heap commit ...

    Note. The above values are non-standard as I have changed them in an attempt to track this down. The problem is, the above settings have no affect upon the outcome.

    I think I've tracked the problem to <perlsources>\win32\vmem.h. Specifically,

    # line 134 VMem::VMem() { m_lRefCount = 1; InitializeCriticalSection(&m_cs); #ifdef _USE_LINKED_LIST m_Dummy.pNext = m_Dummy.pPrev = &m_Dummy; m_Dummy.owner = this; #endif m_hLib = LoadLibrary("msvcrt.dll"); if (m_hLib) { m_pfree = (LPFREE)GetProcAddress(m_hLib, "free"); m_pmalloc = (LPMALLOC)GetProcAddress(m_hLib, "malloc"); m_prealloc = (LPREALLOC)GetProcAddress(m_hLib, "realloc"); } }

    And I think this in the makefile is a contributary factor:

    LIBC = msvcrt.lib

    No proof yet. (I'm on my 3rd perl build; and geez they take a long time!) Just gut feel at this point. What I can say is that the problem still manifests itself when Perl and all its libraries are built with the same compiler (use the same CRT). And that using PERL_MALLOC doesn't change the situation.

    However I get very similar results with a 32 bit Windows system and 5.10.1 btw.

    Thanks for that. I don't suppose you have a AS 5.8.something install kicking around that you could try this on?


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

        That looks like the same underlying problem, but it says in there somewhere a fix was applied to 5.00_3?? But it can still be triggered in current (at least 5.14) builds.

        It also isn't just a problem with MS CRT *alloc(). I tried really hard to trigger the behavior using C, but failed dismally. There is something peculiar about the combination of Perl and the MSCRT *alloc() that fight each other.

        There is also the stuff in win32\vmem.h which looks (to me) like a bastardised C++ class brute-forces into C; but I could never track down whether that stuff is actually used. too many layers of defines and redefines for me to wrap my brain around.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        RIP Neil Armstrong

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://827978]
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (4)
As of 2018-05-22 03:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?