Secondary Display Driver fails to load with /3GB

We are developing a Windows XP 32-bit system with an nVidia card and an ATI
card driving multiple displays, and have encountered a situation where some
of the displays will not become enabled if the /3GB and /USERVA=2500 setting
is used. This problem was first identified using an AMD V5600 from Barco
Medical Imaging using the Barco video driver, and an nVidia FX380. The
results were confirmed to also occur with an AMD Radeon X1800 and an nVidia
FX380 using the latest drivers from AMD and nVidia. The AMD card is driving
two displays, and the nVidia card is driving one display. The problem has
been seen on several PCs.

The specific behavior is as follows: If the PC is booted normally, without
the /3GB or /USERVA=2500 in the BOOT.INI, the system boots, and all three
displays are enabled. But, with the /3GB and /USERVA=2500 settings in the
BOOT.INI, the display on the nVidia card works, but neither display on the
AMD card is operable. There is no change if only one display is attached to
the AMD card. The observation is that something about the /3GB and /USERVA
switches is causing the AMD card to not display an image.

The nVidia card in this setup is in the primary x16 PCIe slot, reported in
WinDBG’s !pci command as being at PCI segment 0, bus 1. The AMD card is in
the secondary x16 PCIe slot, at PCI segment 0, bus 8. However, if the AMD
and nVidia card are swapped while the /3GB and /USERVA=2500 switches are
used, then both displays on the AMD card work, and the one on the nVidia card
is left blank.

Using WinDBG, we have analyzed the boot behavior. We gathered a virtual
memory usage map with !vm, both without and with the /3GB and /USERVA=2500
settings. Those are at the bottom of this message. It was seen that both
cards create an aperture of 256MB. Additionally, no failures were detected
in the AMD driver when memory is allocated or mapped. It was observed that
the AMD miniport is loaded. The AMD DDI is loaded twice, each time it is
loaded the mode list is retrieved, and the DDI is then exited. However, the
DDI is not loaded a third time, when the PDEV would have been created. If we
are interpreting the data correctly, it does not appear that we are running
out of memory or PTEs. But, for some reason, it appears that the OS has
decided not to continue loading the AMD driver.

The question is to identify either how to make this system work, or to
understand why it won’t be able to work. Although switching to cards from a
single vendor, or switching to a 64-bit version of Windows will avoid the
problem, it is far from satisfactory for the customer requirements.

Does anyone know of any condition that would cause the OS to silently
abandon the load of the AMD driver? (As mentioned above, we see the Miniport
load and run normally, all device memory is mapped correctly, and a modest
amount of paged and non-paged pool is allocated, all without any failures or
errors logged.)

What situations, if any, should we be looking for to identify whether/why
the OS is silently abandoning the DDI load? Which tools or other data might
be useful in identifying what is causing the AMD driver to not fully load?

Thanks for any help you can offer.

********** !VM WITHOUT 3GB or USERVA, RADEON + NVIDIA - WORKS
******************

*** Virtual Memory Usage ***
Physical Memory: 847689 ( 3390756 Kb)
Page File: \??\C:\pagefile.sys
Current: 262144 Kb Free Space: 253392 Kb
Minimum: 262144 Kb Maximum: 4190208 Kb
Available Pages: 766345 ( 3065380 Kb)
ResAvail Pages: 790714 ( 3162856 Kb)
Locked IO Pages: 467 ( 1868 Kb)
Free System PTEs: 122163 ( 488652 Kb)

******* 12 system PTE allocations have failed ******

Free NP PTEs: 6246 ( 24984 Kb)
Free Special NP: 99112 ( 396448 Kb)
Modified Pages: 12937 ( 51748 Kb)
Modified PF Pages: 12937 ( 51748 Kb)
NonPagedPool Usage: 5085 ( 20340 Kb)
NonPagedPool Max: 38564 ( 154256 Kb)
PagedPool 0 Usage: 4462 ( 17848 Kb)
PagedPool 1 Usage: 299 ( 1196 Kb)
PagedPool 2 Usage: 306 ( 1224 Kb)
PagedPool 3 Usage: 304 ( 1216 Kb)
PagedPool 4 Usage: 303 ( 1212 Kb)
PagedPool Usage: 5674 ( 22696 Kb)
PagedPool Maximum: 40448 ( 161792 Kb)
Shared Commit: 1060 ( 4240 Kb)
Special Pool: 1020 ( 4080 Kb)
Shared Process: 2140 ( 8560 Kb)
PagedPool Commit: 5674 ( 22696 Kb)
Driver Commit: 4136 ( 16544 Kb)
Committed pages: 83914 ( 335656 Kb)
Commit limit: 877947 ( 3511788 Kb)


********** !VM WITH 3GB and USERVA=2500, RADEON + NVIDIA - FAILS
******************

*** Virtual Memory Usage ***
Physical Memory: 847689 ( 3390756 Kb)
Page File: \??\C:\pagefile.sys
Current: 262144 Kb Free Space: 253392 Kb
Minimum: 262144 Kb Maximum: 4190208 Kb
Available Pages: 766345 ( 3065380 Kb)
ResAvail Pages: 790714 ( 3162856 Kb)
Locked IO Pages: 467 ( 1868 Kb)
Free System PTEs: 122163 ( 488652 Kb)

******* 12 system PTE allocations have failed ******

Free NP PTEs: 6246 ( 24984 Kb)
Free Special NP: 99112 ( 396448 Kb)
Modified Pages: 12937 ( 51748 Kb)
Modified PF Pages: 12937 ( 51748 Kb)
NonPagedPool Usage: 5085 ( 20340 Kb)
NonPagedPool Max: 38564 ( 154256 Kb)
PagedPool 0 Usage: 4462 ( 17848 Kb)
PagedPool 1 Usage: 299 ( 1196 Kb)
PagedPool 2 Usage: 306 ( 1224 Kb)
PagedPool 3 Usage: 304 ( 1216 Kb)
PagedPool 4 Usage: 303 ( 1212 Kb)
PagedPool Usage: 5674 ( 22696 Kb)
PagedPool Maximum: 40448 ( 161792 Kb)
Shared Commit: 1060 ( 4240 Kb)
Special Pool: 1020 ( 4080 Kb)
Shared Process: 2140 ( 8560 Kb)
PagedPool Commit: 5674 ( 22696 Kb)
Driver Commit: 4136 ( 16544 Kb)
Committed pages: 83914 ( 335656 Kb)
Commit limit: 877947 ( 3511788 Kb)

--
JHaggerty

JHaggerty <> wrote:
>
>...
>The specific behavior is as follows: If the PC is booted normally, without
>the /3GB or /USERVA=2500 in the BOOT.INI, the system boots, and all three
>displays are enabled. But, with the /3GB and /USERVA=2500 settings in the
>BOOT.INI, the display on the nVidia card works, but neither display on the
>AMD card is operable. There is no change if only one display is attached to
>the AMD card. The observation is that something about the /3GB and /USERVA
>switches is causing the AMD card to not display an image.
>...
>Using WinDBG, we have analyzed the boot behavior. We gathered a virtual
>memory usage map with !vm, both without and with the /3GB and /USERVA=2500
>settings. Those are at the bottom of this message. It was seen that both
>cards create an aperture of 256MB.

Yep, that'll do it. By using /3GB, you are reducing the amount of kernel
space, which in turns reduces the amount of virtual space available for
non-paged pool and mappings. It's fairly intuitive, isn't it? You have a
TOTAL of 1GB of kernel virtual address space. After chewing up space for
drivers, page tables, and system data structures, you don't have enough
space for two 256MB mappings.

>The question is to identify either how to make this system work, or to
>understand why it won’t be able to work.

Your choices are not simple. Turn off /3GB, or use a 64-bit operating
operating system.
--
Tim Roberts,
Providenza & Boekelheide, Inc.

512M (out of 1G available for the kernel) for video apertures.

Nothing strange that the thing fails.

> The question is to identify either how to make this system work, or to
> understand why it won’t be able to work.

Some subtle condition within one of the drivers.

>Although switching to cards from a
> single vendor, or switching to a 64-bit version of Windows will avoid the
> problem, it is far from satisfactory for the customer requirements.

What about getting rid of /3GB? or upgrading to Win7?

You're pushing the system to its limits, and it starts showing teeth. Nothing strange.

--
Maxim S. Shatskih
Windows DDK MVP

http://www.storagecraft.com

>
>
> It was seen that both cards create an aperture of 256MB.
>
I have to laugh. I quote from Daniel Rutter's infamous web-log entry of
2007:

> Power users with a hankerin' for dual graphics cards may be
> experiencing something of a sinking feeling, at this juncture. Yes,
> the 256Mb reserved for my little old graphics card means exactly what
> you think it means: Those two 768Mb graphics cards you
> can totally justify buying will eat one point five gigabytes of your
> 32-bit memory map all by themselves, cutting you down to a 2.5Gb
> ceiling before you even take the other reservations into account.
>
Go to Dan's Data, look up the "What's with the 3Gb memory barrier?"
article, read, learn, and experience the sinking feeling. There Ain't
No Such Thing as a Zero-Size 256MiB Frame Buffer. Yet people still want
to add 256MiB+256MiB+256MiB and get 0MiB. (-: