Lock pages in physical memory

Discussion in 'Windows Vista Drivers' started by Brett, Jul 10, 2003.

  1. Brett

    Brett Guest

    I need to allocate an enormous amount of physical memory
    (>80% of system memory) and ensure that is is locked into
    physical memory.
    I've considered MmProbeAndLockPages, but if the allocated
    memory is seriously fragmented, the MDL may end up being
    too large.
    Is there another way to ensure that memory allocated from
    the PagedPool never gets paged?
    -Brett
     
    Brett, Jul 10, 2003
    #1
    1. Advertisements

  2. Brett

    Brett Guest

    I'm developing a Windows-based memory test and I need to
    allocate as much memory as possible (I meant >80% of
    AVAILABLE system memory, sorry). Thus, I need to ensure
    that the algorithms used to test memory don't get mixed
    up with paging operations. That is why I want to lock
    pages in memory.
    -Brett
     
    Brett, Jul 14, 2003
    #2
    1. Advertisements

  3. Sorry, but in the virtual-addressing world of WinXP and quite a few
    other OSes, this is not a well-formed question.

    Consider writing your program to get the maximum amount of virtual
    memory. Remove non-essential applicatiosn for the system. Run your
    program. That's about the best you can do without going into kernel
    mode, and it ought to be a decent test.
     
    James Antognini, Jul 14, 2003
    #3
  4. Brett

    Tim Roberts Guest

    That is an incredibly bad idea; there is so much of the memory you can't
    test that it would be essentially worthless. The best memory tests take
    over the machine through a boot floppy, so that they have 100% control over
    100% of the memory. There are several excellent and simple-to-use DOS
    extenders available for free download that will allow you do this is a very
    useful manner..
     
    Tim Roberts, Jul 15, 2003
    #4
  5. Yes, I've even seen memory test that relocate themselves into the VGA
    framebuffer memory and run from there so they can actually test all the
    on-board memory.
     
    Ralf Buschmann, Jul 15, 2003
    #5
  6. Brett

    Brett Guest

    I am fully aware of all the issues associated with
    building a Windows-based memory test. I have worked on a
    proprietary DOS-based memory test for quite some time
    now. What I really need is an answer to my question about
    locking pages in physical memory. If someone has an
    answer, I would appreciate a reply.
    Thank you,
    -Brett
     
    Brett, Jul 15, 2003
    #6
  7. Brett

    Brett Guest

    Yeah, I knew about that test, too. The reason I want to
    do it completely in Windows is because there are certain
    memory events (like power management events) that are
    only present in the Windows environment and often
    transparent to the user (depending on chipset
    implementation). I want to be able to test memory while
    operating under the Windows environment.

    In the end, I am going to do this and I am going to do it
    in Windows.

    So, what I'm really after is a way to force pages to lock
    into physical memory so I can test them. If anyone has a
    solution to this problem, please post it here or email me
    directly.
    Thank you,
    -Brett
     
    Brett, Jul 16, 2003
    #7
  8. Brett

    Brett Guest

    I do want to mention that I appreciate all who have
    posted on this now lengthy thread. I understand that my
    request is quite extraordinary and that there are similar
    ways of doing it easier. But, alas, I'm bound to do it
    the way I've mentioned (in Windows). I'm still
    solutionless, so if you can offer help, let me know.
    Thanks again,
    Brett
     
    Brett, Jul 16, 2003
    #8
  9. This newsgroup is concerned with the kernel, and one ought to expect
    answers that exploit the kernel. Lucky for you, the kernel is the way to
    go, and you've been pointed in the direction of using kernel function.
    You've been told that's the way to exercise some control over things
    that concern you. Since you seem unreceptive to that advice, why keep
    asking?
     
    James Antognini, Jul 16, 2003
    #9
  10. Brett

    Brian Catlin Guest

    [...snip...]
    OK, if you're determined to go this route..

    WARNING THE FOLLOWING IS NOT FOR THE SQUEAMISH, AND WILL MORE THAN LIKELY
    CORRUPT YOUR SYSTEM

    First there are a few issues that need to be addressed:
    1. Not all physical memory is pageable, so how are you going to test those
    pages? (I'm assuming that your memory test is destructive (of the memory
    contents) and somewhat time consuming).
    To get around this, you would have to essentially stop processing on the system,
    copy the contents of the page to test, and then change the page table entry to
    point to the new page. To stop processing, I would send DPCs to all the CPUs,
    and have the DPC set its IRQL to 31, and then spin waiting for all the rest of
    the CPUs to do the same thing. When all the CPUs are spinning, then one of the
    CPUs should be able to copy the contents of the page into another page (that
    you allocated *before* raising to IRQL 31), and then go diddle the page tables
    (Landy, don't shoot me!), and then flush the TLB. You could probably use this
    technique to run through all of memory.

    2. Not all physical memory is accessible (if I remember correctly, the disk
    boot/crash dump driver is unmapped while the system is running, to prevent it
    from being corrupted)
    You'll have to allocate some system page table entries, and then fill them in to
    point to the inaccessible pages (see the technique above for making the system
    quiescent), and then flush the TLB

    You're going to have to spend more than a little time in the debugger and the
    Intel documentation to figure out the page tables and page frame data structures
    layout. Also, remember that the page tables themselves are pageable.

    Good luck - you'll need it. By the way, what company do you work for? I want
    to make sure I don't mistakenly buy something from them that has this code in
    it.

    -Brian

    Brian Catlin, Sannas Consulting 310-798-8930
    Windows Network, Video, WDM Device Driver Training & Consulting
    See WWW.AZIUS.COM for courses and scheduling
     
    Brian Catlin, Jul 16, 2003
    #10
  11. Before doing any of the horrible thing recomenede below,
    give a look at the output of the debugger command '!pfn 0 8000000F' on KD.
    This will give you an idea of which physical pages are used by
    the os at the given moment, and which ones are availalbe for you to run
    your memory test.
    Before you put to spin all the CPUs besides the one you are wishing to use
    as a worker,
    make sure the code they are spinning on is not going to be touched by your
    memory test.
    You may argue that the local CPU cache would allow CPUs to spin even if the
    physical memory is pulled from underneath them, but you have to be really
    careful
    on what are the side effects of invalidating caches and TLBs across all the
    CPUs
    your test will run on.
    On the side, if the system is using some megs
    of memory for the current working set of the running processes,
    you will have to carefully save the page away before testing the undelaying
    physical page.
    If you can do that one page at time, you may just need one unused physical
    page.
    Naturally you will have to skip the region of physical memory used by
    devices to perform DMA,
    unless you disable all the DMA devices before running your test.
    If you choose not to pay attention to all these details,
    you can limit your test to all the physical pages not currently in use by
    the OS
    (or reserved by the OS for special purposes)
    after you've put the system 'to sleep'.

    Honestly, a memory checker is better developed in a minimal bootstrap code,
    after the CPU is in protected mode with no paging enabled and no TLB/caching
    enabled.
    A minimal bootstrap code can be written in less that 20 assembly instruction
    (if you target x86),
    and then you can jump to your C/C++ code.

    --
    This posting is provided "AS IS" with no warranties, and confers no rights.
    Use of any included script samples are subject to the terms specified at
    http://www.microsoft.com/info/cpyright.htm


     
    Ivan Brugiolo [MSFT], Jul 16, 2003
    #11
  12. Brett

    Brett Guest

    I appreciate the post and your solution is intriguing, but
    I am not trying to test all of memory, just as much as I
    can get my hands on. The memory errors I'm concerned about
    are usually algorithm dependent rather than specific bit
    fails. Getting as much memory as possible is best, but
    getting the entire array is overkill.

    And as for this...
    HaHaHa - I can assure you that you'll never buy software
    from my company, I'd be afraid to give it to the world and
    have to support it.
    This is an "Internal Use Only" type of thing.
     
    Brett, Jul 16, 2003
    #12
  13. Brett

    Brett Guest

    I can appreciate that this has to be done with the kernel.
    That is why I posted this here in the first place. Now, if
    you would kindly give me a direction of how to do this
    with kernel functions...
    -Brett
     
    Brett, Jul 16, 2003
    #13
  14. Brett

    Brian Catlin Guest

    In that case, just grab huge chunks of paged pool and then lock down the pages

    -Brian

    Brian Catlin, Sannas Consulting 310-798-8930
    Windows Network, Video, WDM Device Driver Training & Consulting
    See WWW.AZIUS.COM for courses and scheduling
     
    Brian Catlin, Jul 16, 2003
    #14
  15. Brett

    Brett Guest

    In that case, just grab huge chunks of paged pool and
    This is essentially what I am asking. Back to the start of
    this thread... I want to lock down the pages, but I'm not
    sure what else I can use besides MmProbeAndLockPages. I
    would use it, but I'm afraid that a heavily fragmented
    memory chunk will cause the MDL to be too large and
    therefore fail. Isn't there a limit to how much
    MmProbeAndLockPages will actually lock?
     
    Brett, Jul 16, 2003
    #15
  16. Brett

    Nick Ryan Guest

    Fragmentation is not an issue here. Even a large chunk of virtual memory
    corresponding to a continuous piece of physical memory still consumes
    one page-table entry per 4KB page (in some cases 4MB pages are used by
    Windows internally but you can't make use of them yourself).

    MmProbeAndLockPages should not fail assuming you have enough physical
    memory to back the range you want to make resident.
    MmGetSystemAddressForMdlSafe is what can fail more often, since it
    consumes system PTEs, but you should not need to call this since you
    already have a virtual address for the page range you're working with
    (assuming you allocated the pages in your user-mode app).

    But unless you're picking out physical ranges to test in a systematic
    way (using Brian and Ivan's techniques), at best you're only going to be
    able to test a small portion of available physical memory (i.e. the size
    of the largest block of memory for which MmProbeAndLockPages will work).
     
    Nick Ryan, Jul 17, 2003
    #16
  17. Brett

    Nick Ryan Guest

    Fragmentation is not an issue here. Even a large chunk of virtual memory
    corresponding to a continuous piece of physical memory still consumes
    one page-table entry per 4KB page (in some cases 4MB pages are used by
    Windows internally but you can't make use of them yourself).

    MmProbeAndLockPages should not fail assuming you have enough physical
    memory to back the range you want to make resident.
    MmGetSystemAddressForMdlSafe is what can fail more often, since it
    consumes system PTEs, but you should not need to call this since you
    already have a virtual address for the page range you're working with
    (assuming you allocated the pages in your user-mode app).

    But unless you're picking out physical ranges to test in a systematic
    way (using Brian and Ivan's techniques), at best you're only going to be
    able to test a small portion of available physical memory (i.e. the size
    of the largest block of memory for which MmProbeAndLockPages will work).
     
    Nick Ryan, Jul 17, 2003
    #17
  18. Brett

    Nick Ryan Guest

    Sorry guys, you may see a couple of dupes from me today on some
    messages. Comcast's network has basically been rebooting itself all day
    long, so some of my attempts to resend may have gotten stuck in the
    queue for a few hours.
     
    Nick Ryan, Jul 17, 2003
    #18
  19. Why not just run an user mode test? If you don't request too much of the
    memory, you can expect it won't be paged out while your test is running.
    See, for example my memory test MEMTEST at
    http://home.earthlink.net/~alegr/download/memtest.htm

    MEMTEST can run under both bare metal environment and Windows. When run
    under Windows, it can also run I/O test in parallel. This feature helped me
    to detect instability in one of my computers: the memory would only be
    corrupted if DMA disk transfers were running, and it was NOT that I/O buffer
    memory! Critical OS structures were corrupted so that bit 0x80000000 was
    occasionally getting set! There are also some add-on IDE controllers that
    suffer from data corruption when two IDE channels work in parallel at speed
    exeeeding PCI throughput.
     
    Alexander Grigoriev, Jul 17, 2003
    #19
  20. Brett

    Brett Guest

    Okay, so back to my main question:
    Is there any way in either kernel mode or user mode to
    allocate a large amount (80%-100%) of AVAILABLE physical
    memory (meaning PagedPool only) and lock it into physical
    memory so it does not get paged out?
    I'm not trying to get anyone to write my program for me,
    but I'm definitely stuck and I need a clear direction.
    Thanks for all the help,
    -Brett
     
    Brett, Jul 17, 2003
    #20
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.