Paging

paging!

hey!
although i finally figured out how memory descriptors lists work in theory,
thanks to this article: http://www.osronline.com/article.cfm?id=423
i dis assembled the MmBuildForNonPagedPool function anyways for the sake of
interest , i found the
logic used to get the physical address a bit confusing and out of sync from
my knowledge of paging.

mov ecx,edx
shr ecx,12
and ecx,3FF8
mov edi,[ecx+C0600000]
nov ecx,[ecx+C0600004]

the above is an extract from the MmBuildForNonPagedPool function. the ecx
register initially stores Mdl.startVa
which is the starting virtual address of the buffer described by the
MDL(aligned to 4096 byte boundry or page frame size boundry).
in the second instruction the ecx register is shifted right 18 times, which
gives us the rightmost 14 bits of virtual address
i.e bits (31:18) the third instruction removes 3 leftmost bits. This implies
that the function multiplies rightmost 11 bits of the virutal address
with 8 (8 is the size of a page directory entry with PAE bit (5) of the CR4
register enabled) for example if we started with a
virtual address of say 81274567 then.. :

1000 0001 0010 0111 0100 0101 0110 0111 = 81274567 , starting virual address
0000 0000 0000 0000 0010 0000 0100 1001 = 00002049 , result after shifting
18 times
0000 0000 0000 0000 0010 0000 0100 1000 = 00002048 , after stripping the
last 3 bits which is the same as
1000 0001 001 (this first 11 bits of the VA) * 8 = 10 0000 0100 1000

this number (00002048 in our example) is used to index the page directory
(C0600000, page directory base virtual address when
PAE is set) notice that we fetch 8 bits from the page directory, 4 of them
are stored in edi and the other 4 are stored in ecx
Question1: which paging mechanism uses 11 bits to index a page directory ?
according to the intel manuals when PAE is on
2 bits are used to index the PDPT and 9 bits are used to index the page
directory.

here is the next set of asm instructions from the MmBuildForNonPagedPool
function :

mov [ebp-4],ecx ; store the higher 4 bytes of the PDE on the stack as ecx
is going to used in the following instructions
mov ecx,00000081; this clears all bits in ecx except bits 0 and 7
mov [ebp-8],edi ; store the lower 4 bytes of the PDE on the stack as edi is
going to used in the following instructions
and edi,ecx;
cmp edi,ecx; compare the bits 0 and bits 7 of the PDE
jnz 8050A00;
the jump is taken if bits 0 and 7 of the PDE are set, now it makes sense why
its checking for bit 0 as it indicates
wether the PDE is paged in or paged out, but the bit 7 according to the
intel manuals is set to 0 always then why bother
checking it? that is Question 2.
the MmBuildForNonPagedPool function is used when the physical pages
described by the virtual address buffer described by the MDL
is in non paged pool i.e the caller is sure that all the phyical pages
associated with the virutall address buffer are in nonpaged pool, so i am
guessing the jnz instruction is a check to insure that the PDE is paged in
(as page tables can be paged out) but i am not sure why its checking
for bit 7 :S

to confuse matters further :
once the jump is taken i.e the PDE is paged in :

SHR EDX,09 ; edx initially contains the mdl.startVa,
AND EDX,007FFFF8; the above 2 instructions multiply the rightmost 20 bits of
the Virtual address with 8
SUB EDX,40000000; is this compiler optimization? i dont understand how this
is better than, add EDX,C0000000??
anyways the purpose of this instruction is to index the Page table (sub
Question: according to the
intel manuals the page table base is obtained from the PDE? why is it
assumed that the Page table
base is always C0000000??)
mov ecx,[edx]; store the lower 4 bytes of the PTE in ecx
mov edi,[edx+04]; store the higher 4 bytes of the PTE in edi
SHRD ecx,edi,0C; shifts the 8 byte PTE 12 times to the right, its looking
for the page address base
AND ecx,03FFFFFF; its keeping 26 bits and stripping of the rest hmm
according to the manuals the page address base is only 24 bits
then why 26 bits? (this value is stored in the array below the MDL header so
i guess its the page address base) that is Question 4

thanks for your help

codeFather <> wrote:
>
>paging!
>...
>Question1: which paging mechanism uses 11 bits to index a page directory ?
>according to the intel manuals when PAE is on
>2 bits are used to index the PDPT and 9 bits are used to index the page
>directory.

Windows stores all of the page table directories sequentially. So, even
though the 11 bits happens to span several directories, they know the fetch
will work.

>the jump is taken if bits 0 and 7 of the PDE are set, now it makes sense why
>its checking for bit 0 as it indicates
>wether the PDE is paged in or paged out, but the bit 7 according to the
>intel manuals is set to 0 always then why bother
>checking it? that is Question 2.

What manual are you looking at? Bit 7 of the PDE is the large/small page
bit. 0 means it's a 4k page. 1 means it's a 4M page.

>Question: according to the
>intel manuals the page table base is obtained from the PDE? why is it
>assumed that the Page table base is always C0000000??)

Because the Windows kernel created the page tables do begin with. Like the
PDEs, Windows creates the page tables sequentially. That makes it possible
to access them as if they were all one big table.

>mov ecx,[edx]; store the lower 4 bytes of the PTE in ecx
>mov edi,[edx+04]; store the higher 4 bytes of the PTE in edi
>SHRD ecx,edi,0C; shifts the 8 byte PTE 12 times to the right, its looking
>for the page address base
>AND ecx,03FFFFFF; its keeping 26 bits and stripping of the rest hmm
>according to the manuals the page address base is only 24 bits
>then why 26 bits? (this value is stored in the array below the MDL header so
>i guess its the page address base) that is Question 4

It should be clear that 24 bits is not nearly enough for a physical page
base.
--
Tim Roberts,
Providenza & Boekelheide, Inc.

thanks for the post

>Windows stores all of the page table directories sequentially. So, even
>though the 11 bits happens to span several directories, they know the fetch
>will work.

"sequentially" hmmm. do you mean to say that by using 11 bits what we are
actually doing is taking the rightmost 2 bits and multiplying them by the
size of a page directory (which is Math.pow(2, 11-2) = 512) and to index
the 8 byte PDE all we need to do is change the 9 bits (following the 2 bits
which index the PDPT) for example if the first 2 bits were 10 then
10 : first 2 bits used to index the PDPT
10 000000000 : append 9 bits i.e multiplying the first 2 bits with 512 (size
of a page directory)
10 000000000 000 : append 3 bits i.e multiplying the whole by 8 (size of the
PDE)

10 000000000 000 : the first page directory entry.
10 000000001 000 : the second page directory entry.
10 000000010 000 : the third page directory entry and so on.....

and i guess similar reasoning applies for page table entries, there we use
the first 20 bits and multiply them by 8 =>
10 000000001 : 11 bits for the Page directory index
10 000000001 000000000 : multiplyed by 512 (size of the Page table)
10 000000001 000000000 000 : multiplyed by 8 (size of the page table entry)

>What manual are you looking at? Bit 7 of the PDE is the large/small page
>bit. 0 means it's a 4k page. 1 means it's a 4M page.

my bad. i did not look carefully, actually i assumed that every important
bit would be labelled anyways when i rechecked the manual i found bit 7 to be
clear for 4 byte pages when PAE is enabled and bit 7 set for 4 MB pages when
PAE is enabled (it is the Page Size bit, its labelled in figure 3-15,
3.6.4), i am referring to the (Intel Architecture Software Developer’s Manual
Volume 3: System Programming (Order Number 243190) heres the download link
http://www.intel.com/design/pentiumi...als/243190.htm) chapter 3.6
(paging) section 8 i.e 3.6.8 (physical address extension)

>It should be clear that 24 bits is not nearly enough for a physical page
>base.

the only thing i do not still understand is the need for a 26 bit address
base?since the page size 4KB (clear from the fact the JNZ instruction took
the jump and bit 0 was set) we need 12 bits to index every byte. But if we
use 26 bits as page address base then we only get (36-26) = 10 bits to index
the 4KB page, but with 10 bits we can only index 2^10 = 1024 bytes = 1KB. On
the other hand if we use 24 bits then we get (36-24) = 12 bits to index the
4KB page, which is what we need. And the same is mentioned in (Figure 3-20.
Format of Page-Directory-Pointer-Table, Page-Directory, and Page-Table
Entries for 4-KByte Pages and 36-Bit Extended Physical Addresses) of the
intel manual. 96th page of the .pdf,

thanks again!

codeFather <> wrote:
>
>the only thing i do not still understand is the need for a 26 bit address
>base?since the page size 4KB (clear from the fact the JNZ instruction took
>the jump and bit 0 was set) we need 12 bits to index every byte. But if we
>use 26 bits as page address base then we only get (36-26) = 10 bits to index
>the 4KB page, but with 10 bits we can only index 2^10 = 1024 bytes = 1KB. On
>the other hand if we use 24 bits then we get (36-24) = 12 bits to index the
>4KB page, which is what we need.

Which operating system are you using? Even with PAE, XP and Vista will not
allow access to any memory beyond the 4GB mark, because of a licensing
restriction. It's possible that's related here, but I'm not sure.

You'd have to check Server 2003 or Server 2008 to see how they handle it.
--
Tim Roberts,
Providenza & Boekelheide, Inc.

i am using windows XP service pack 2. But if access to only 4GB mark is
allowed, then why waste clock cycles fetching 2 4byte entries, the paging
mechanism without Physical Address Extension would be more efficient then,
right?

codeFather <> wrote:
>
>i am using windows XP service pack 2. But if access to only 4GB mark is
>allowed, then why waste clock cycles fetching 2 4byte entries, the paging
>mechanism without Physical Address Extension would be more efficient then,
>right?

The 4GB limit is strictly a software thing.

And remember that page table entries are aggressively cached in the TLB. It
doesn't take any extra clock cycles to fetch two 4-byte entries.
--
Tim Roberts,
Providenza & Boekelheide, Inc.