How do I dig deeper into a crash dump with WinDbg to idenify locku

Using KMDF 1.7 to update an OLD NT 4.0 driver that worked on Win2000 sp4 to
now work on Win XP sp3, I was able to get the driver to the testing phase in
6 weeks while learning the KMDF (the KMDF is one huge step forward in driver
development). In very rare instances, the driver locks up for users signed
in as administrators, but the lock-up is easily reproduced in less than 1/2
hour. We force a blue screen (Ctrl, Scroll, Scroll) and sure enough it is
frozen in the DPC after getting the first spinlock and hung waiting for the
second spinlock.

Looks like a classical getting-locks-out-of-order system freeze. For the
last 3 days I have studied the code and don't see the cause of the lock-up.

****Question***** How can I use WinDbg to find and look into the data
sctructures stored in my driver to help me determine who might be holding the
other lock? The driver originally used KSPIN_LOCKs, and I have crash dumps
for that. The lastest test versions (shots in the dark) use WDFSPINLOCKs
(was hoping I would get some WDF warning somewhere, wishful thinking).

More details: The driver is for an audio interface card with three ports,
two for headsets one for 4-wire trunk. The driver uses IOCTLs heavily to
control the DSP on the board. There are three kernel threads (one per
channel/audio interface). The DLL sends IOCTLs to 1) send commands to the
DSP's three channels, 2) change states of the driver, 3) poll the driver for
events collected from the DSP. The locks keep the DPC and the channel
threads and the IOCTLS from using the 1) DSP interface and 2) channel state
data simultaneously. Most locks are held for just a few assignment
statements (atomic state changes) or for sending a command or getting a
response from the DSP. The DPC acquires the two locks at the very beginning
A then B, then releases them B then A at the end.

Thanks in advance, John Bond

Have you looked at the other processors in the system to see if you can find
the CPU holding the spinlock? "!running -t" is useful here.

If there's no other CPU out there doing something with the lock held then
you likely have an error path that does not drop the lock correctly. You can
also try !wdfkd.wdfspinlock on the lock handle to try to get some debugging
info (I've never used this so I'm not sure what kind of information it
outputs).

-scott

--
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com




"John Bond" <> wrote in message
news:56EDAF0C-777F-42F4-A705-...
> Using KMDF 1.7 to update an OLD NT 4.0 driver that worked on Win2000 sp4
> to
> now work on Win XP sp3, I was able to get the driver to the testing phase
> in
> 6 weeks while learning the KMDF (the KMDF is one huge step forward in
> driver
> development). In very rare instances, the driver locks up for users
> signed
> in as administrators, but the lock-up is easily reproduced in less than
> 1/2
> hour. We force a blue screen (Ctrl, Scroll, Scroll) and sure enough it is
> frozen in the DPC after getting the first spinlock and hung waiting for
> the
> second spinlock.
>
> Looks like a classical getting-locks-out-of-order system freeze. For the
> last 3 days I have studied the code and don't see the cause of the
> lock-up.
>
> ****Question***** How can I use WinDbg to find and look into the data
> sctructures stored in my driver to help me determine who might be holding
> the
> other lock? The driver originally used KSPIN_LOCKs, and I have crash
> dumps
> for that. The lastest test versions (shots in the dark) use WDFSPINLOCKs
> (was hoping I would get some WDF warning somewhere, wishful thinking).
>
> More details: The driver is for an audio interface card with three ports,
> two for headsets one for 4-wire trunk. The driver uses IOCTLs heavily to
> control the DSP on the board. There are three kernel threads (one per
> channel/audio interface). The DLL sends IOCTLs to 1) send commands to the
> DSP's three channels, 2) change states of the driver, 3) poll the driver
> for
> events collected from the DSP. The locks keep the DPC and the channel
> threads and the IOCTLS from using the 1) DSP interface and 2) channel
> state
> data simultaneously. Most locks are held for just a few assignment
> statements (atomic state changes) or for sending a command or getting a
> response from the DSP. The DPC acquires the two locks at the very
> beginning
> A then B, then releases them B then A at the end.
>
> Thanks in advance, John Bond
>

> response from the DSP. The DPC acquires the two locks at the very beginning
> A then B, then releases them B then A at the end.

Try to rewrite the code to use 1 and only 1 lock.

--
Maxim S. Shatskih
Windows DDK MVP

http://www.storagecraft.com

Max, that sounded like a good idea, but it quickly failed and it was clear
that since the lock was used twice in some places, it was tough to "get it
right" easily. But in general that is a very good idea. JB

"Maxim S. Shatskih" wrote:

> > response from the DSP. The DPC acquires the two locks at the very beginning
> > A then B, then releases them B then A at the end.
>
> Try to rewrite the code to use 1 and only 1 lock.
>
> --
> Maxim S. Shatskih
> Windows DDK MVP
>
> http://www.storagecraft.com
>
> .
>

Scott, thanks! I realized that I had not yet converted to WDFSPINLOCKs so
the captures I get today will let me use !wdfkd.wdfspinlock.

Question about "!running -t", it told me there were three processors and all
were idle for a hyperthreaded processor crash dump... I took the dump off the
crashed computer and was looking at it on my laptop. Could it have been
counting the laptop's cpu? Why did it say that all three were idle, was that
due to the Ctrl-Scroll-Scroll induced crash?

-JB

"Scott Noone" wrote:

> Have you looked at the other processors in the system to see if you can find
> the CPU holding the spinlock? "!running -t" is useful here.
>
> If there's no other CPU out there doing something with the lock held then
> you likely have an error path that does not drop the lock correctly. You can
> also try !wdfkd.wdfspinlock on the lock handle to try to get some debugging
> info (I've never used this so I'm not sure what kind of information it
> outputs).
>
> -scott
>
> --
> Scott Noone
> Consulting Associate
> OSR Open Systems Resources, Inc.
> http://www.osronline.com
>
>
>
>
> "John Bond" <> wrote in message
> news:56EDAF0C-777F-42F4-A705-...
> > Using KMDF 1.7 to update an OLD NT 4.0 driver that worked on Win2000 sp4
> > to
> > now work on Win XP sp3, I was able to get the driver to the testing phase
> > in
> > 6 weeks while learning the KMDF (the KMDF is one huge step forward in
> > driver
> > development). In very rare instances, the driver locks up for users
> > signed
> > in as administrators, but the lock-up is easily reproduced in less than
> > 1/2
> > hour. We force a blue screen (Ctrl, Scroll, Scroll) and sure enough it is
> > frozen in the DPC after getting the first spinlock and hung waiting for
> > the
> > second spinlock.
> >
> > Looks like a classical getting-locks-out-of-order system freeze. For the
> > last 3 days I have studied the code and don't see the cause of the
> > lock-up.
> >
> > ****Question***** How can I use WinDbg to find and look into the data
> > sctructures stored in my driver to help me determine who might be holding
> > the
> > other lock? The driver originally used KSPIN_LOCKs, and I have crash
> > dumps
> > for that. The lastest test versions (shots in the dark) use WDFSPINLOCKs
> > (was hoping I would get some WDF warning somewhere, wishful thinking).
> >
> > More details: The driver is for an audio interface card with three ports,
> > two for headsets one for 4-wire trunk. The driver uses IOCTLs heavily to
> > control the DSP on the board. There are three kernel threads (one per
> > channel/audio interface). The DLL sends IOCTLs to 1) send commands to the
> > DSP's three channels, 2) change states of the driver, 3) poll the driver
> > for
> > events collected from the DSP. The locks keep the DPC and the channel
> > threads and the IOCTLS from using the 1) DSP interface and 2) channel
> > state
> > data simultaneously. Most locks are held for just a few assignment
> > statements (atomic state changes) or for sending a command or getting a
> > response from the DSP. The DPC acquires the two locks at the very
> > beginning
> > A then B, then releases them B then A at the end.
> >
> > Thanks in advance, John Bond
> >
>
>
> .
>

> Question about "!running -t", it told me there were three processors and
> all
> were idle for a hyperthreaded processor crash dump

I suspect that it really showed three idle and one running (quad proc,
right?). That means that you don't likely have a race here since you're
deadlocking on a single CPU.

I'd start looking for error paths that don't drop the spinlock, since
they're not recursively acquireable a dangling lock can deadlock like this.
!wdfspinlock might be helpful here.

(And the advice of using fewer locks certainly makes things simpler, but in
a world where Windows supports 256 processors breaking locks up is sometimes
necessary for high perf!)

-scott

--
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com


"John Bond" <> wrote in message
news:6C684507-6473-4AAF-841C-...
> Scott, thanks! I realized that I had not yet converted to WDFSPINLOCKs so
> the captures I get today will let me use !wdfkd.wdfspinlock.
>
> Question about "!running -t", it told me there were three processors and
> all
> were idle for a hyperthreaded processor crash dump... I took the dump off
> the
> crashed computer and was looking at it on my laptop. Could it have been
> counting the laptop's cpu? Why did it say that all three were idle, was
> that
> due to the Ctrl-Scroll-Scroll induced crash?
>
> -JB
>
> "Scott Noone" wrote:
>
>> Have you looked at the other processors in the system to see if you can
>> find
>> the CPU holding the spinlock? "!running -t" is useful here.
>>
>> If there's no other CPU out there doing something with the lock held then
>> you likely have an error path that does not drop the lock correctly. You
>> can
>> also try !wdfkd.wdfspinlock on the lock handle to try to get some
>> debugging
>> info (I've never used this so I'm not sure what kind of information it
>> outputs).
>>
>> -scott
>>
>> --
>> Scott Noone
>> Consulting Associate
>> OSR Open Systems Resources, Inc.
>> http://www.osronline.com
>>
>>
>>
>>
>> "John Bond" <> wrote in message
>> news:56EDAF0C-777F-42F4-A705-...
>> > Using KMDF 1.7 to update an OLD NT 4.0 driver that worked on Win2000
>> > sp4
>> > to
>> > now work on Win XP sp3, I was able to get the driver to the testing
>> > phase
>> > in
>> > 6 weeks while learning the KMDF (the KMDF is one huge step forward in
>> > driver
>> > development). In very rare instances, the driver locks up for users
>> > signed
>> > in as administrators, but the lock-up is easily reproduced in less than
>> > 1/2
>> > hour. We force a blue screen (Ctrl, Scroll, Scroll) and sure enough it
>> > is
>> > frozen in the DPC after getting the first spinlock and hung waiting for
>> > the
>> > second spinlock.
>> >
>> > Looks like a classical getting-locks-out-of-order system freeze. For
>> > the
>> > last 3 days I have studied the code and don't see the cause of the
>> > lock-up.
>> >
>> > ****Question***** How can I use WinDbg to find and look into the data
>> > sctructures stored in my driver to help me determine who might be
>> > holding
>> > the
>> > other lock? The driver originally used KSPIN_LOCKs, and I have crash
>> > dumps
>> > for that. The lastest test versions (shots in the dark) use
>> > WDFSPINLOCKs
>> > (was hoping I would get some WDF warning somewhere, wishful thinking).
>> >
>> > More details: The driver is for an audio interface card with three
>> > ports,
>> > two for headsets one for 4-wire trunk. The driver uses IOCTLs heavily
>> > to
>> > control the DSP on the board. There are three kernel threads (one per
>> > channel/audio interface). The DLL sends IOCTLs to 1) send commands to
>> > the
>> > DSP's three channels, 2) change states of the driver, 3) poll the
>> > driver
>> > for
>> > events collected from the DSP. The locks keep the DPC and the channel
>> > threads and the IOCTLS from using the 1) DSP interface and 2) channel
>> > state
>> > data simultaneously. Most locks are held for just a few assignment
>> > statements (atomic state changes) or for sending a command or getting a
>> > response from the DSP. The DPC acquires the two locks at the very
>> > beginning
>> > A then B, then releases them B then A at the end.
>> >
>> > Thanks in advance, John Bond
>> >
>>
>>
>> .
>>

you should enable for the io veifier for deadlock detection as well as the
KMDF verifier (which will turn on ownership tracking for the WDFSPINLOCK as
well as acquisition / release history)

d

--

This posting is provided "AS IS" with no warranties, and confers no rights.


"Scott Noone" <> wrote in message
news:C2C6E1B5-EEAD-443F-B150-...
>> Question about "!running -t", it told me there were three processors and
>> all
>> were idle for a hyperthreaded processor crash dump
>
> I suspect that it really showed three idle and one running (quad proc,
> right?). That means that you don't likely have a race here since you're
> deadlocking on a single CPU.
>
> I'd start looking for error paths that don't drop the spinlock, since
> they're not recursively acquireable a dangling lock can deadlock like
> this. !wdfspinlock might be helpful here.
>
> (And the advice of using fewer locks certainly makes things simpler, but
> in a world where Windows supports 256 processors breaking locks up is
> sometimes necessary for high perf!)
>
> -scott
>
> --
> Scott Noone
> Consulting Associate
> OSR Open Systems Resources, Inc.
> http://www.osronline.com
>
>
> "John Bond" <> wrote in message
> news:6C684507-6473-4AAF-841C-...
>> Scott, thanks! I realized that I had not yet converted to WDFSPINLOCKs
>> so
>> the captures I get today will let me use !wdfkd.wdfspinlock.
>>
>> Question about "!running -t", it told me there were three processors and
>> all
>> were idle for a hyperthreaded processor crash dump... I took the dump off
>> the
>> crashed computer and was looking at it on my laptop. Could it have been
>> counting the laptop's cpu? Why did it say that all three were idle, was
>> that
>> due to the Ctrl-Scroll-Scroll induced crash?
>>
>> -JB
>>
>> "Scott Noone" wrote:
>>
>>> Have you looked at the other processors in the system to see if you can
>>> find
>>> the CPU holding the spinlock? "!running -t" is useful here.
>>>
>>> If there's no other CPU out there doing something with the lock held
>>> then
>>> you likely have an error path that does not drop the lock correctly. You
>>> can
>>> also try !wdfkd.wdfspinlock on the lock handle to try to get some
>>> debugging
>>> info (I've never used this so I'm not sure what kind of information it
>>> outputs).
>>>
>>> -scott
>>>
>>> --
>>> Scott Noone
>>> Consulting Associate
>>> OSR Open Systems Resources, Inc.
>>> http://www.osronline.com
>>>
>>>
>>>
>>>
>>> "John Bond" <> wrote in message
>>> news:56EDAF0C-777F-42F4-A705-...
>>> > Using KMDF 1.7 to update an OLD NT 4.0 driver that worked on Win2000
>>> > sp4
>>> > to
>>> > now work on Win XP sp3, I was able to get the driver to the testing
>>> > phase
>>> > in
>>> > 6 weeks while learning the KMDF (the KMDF is one huge step forward in
>>> > driver
>>> > development). In very rare instances, the driver locks up for users
>>> > signed
>>> > in as administrators, but the lock-up is easily reproduced in less
>>> > than
>>> > 1/2
>>> > hour. We force a blue screen (Ctrl, Scroll, Scroll) and sure enough
>>> > it is
>>> > frozen in the DPC after getting the first spinlock and hung waiting
>>> > for
>>> > the
>>> > second spinlock.
>>> >
>>> > Looks like a classical getting-locks-out-of-order system freeze. For
>>> > the
>>> > last 3 days I have studied the code and don't see the cause of the
>>> > lock-up.
>>> >
>>> > ****Question***** How can I use WinDbg to find and look into the data
>>> > sctructures stored in my driver to help me determine who might be
>>> > holding
>>> > the
>>> > other lock? The driver originally used KSPIN_LOCKs, and I have crash
>>> > dumps
>>> > for that. The lastest test versions (shots in the dark) use
>>> > WDFSPINLOCKs
>>> > (was hoping I would get some WDF warning somewhere, wishful thinking).
>>> >
>>> > More details: The driver is for an audio interface card with three
>>> > ports,
>>> > two for headsets one for 4-wire trunk. The driver uses IOCTLs heavily
>>> > to
>>> > control the DSP on the board. There are three kernel threads (one per
>>> > channel/audio interface). The DLL sends IOCTLs to 1) send commands to
>>> > the
>>> > DSP's three channels, 2) change states of the driver, 3) poll the
>>> > driver
>>> > for
>>> > events collected from the DSP. The locks keep the DPC and the channel
>>> > threads and the IOCTLS from using the 1) DSP interface and 2) channel
>>> > state
>>> > data simultaneously. Most locks are held for just a few assignment
>>> > statements (atomic state changes) or for sending a command or getting
>>> > a
>>> > response from the DSP. The DPC acquires the two locks at the very
>>> > beginning
>>> > A then B, then releases them B then A at the end.
>>> >
>>> > Thanks in advance, John Bond
>>> >
>>>
>>>
>>> .
>>>

>12. Sometimes it may be tough to know the difference between a
>deadlock and a bus hang; adding debug code or a log file may help.

13. Review the newly written code - better the next day - for _lock use_ only.

--
Maxim S. Shatskih
Windows DDK MVP

http://www.storagecraft.com