receiving packets on multiprocessor machine

When two processor machine with W2K3 is overloaded by high network traffic,
one processor raises almost to 100% and second only 20-30%.
In some mails in this group this problem was discussed and explained that
reason is that interrupt for receiving packets is targeted to one processor.
Also was discussed that processor afinity for DPC is possible to force to
some processor.
My question is if such work has sense to avoid such work if it is crazy idea.
Has sense to try to force some DPC routines to other processor than is
interrupted by receiving packets ?
Or system does load balancing the best how it can be and my forcing of DPC
routines to other processor has not sense in my driver ?

Thx!
Peter

If one CPU is 100% busy handling interrupts, then what you need is to
find some way to implement "interrupt moderation". That is, you should
not have a one-to-one relation between received/sent frames and
interrupts. Most modern network cards/chips allow for interrupt
moderation by e.g. delaying interrupts to some extent and/or counting
frames before issuing an interrupt.

Run perfmon to visualize the number of interrupts and DPCs.

See also:

"Analyze Driver Performance"
http://www.microsoft.com/whdc/driver...m/drvperf.mspx

"High Performance Network Adapters and Drivers"
http://www.microsoft.com/whdc/device...ters-Drvs.mspx

"Windows Scalable Networking Initiative"
http://www.microsoft.com/whdc/device/network/scale.mspx

Stephan
---
Peter wrote:
> When two processor machine with W2K3 is overloaded by high network traffic,
> one processor raises almost to 100% and second only 20-30%.
> In some mails in this group this problem was discussed and explained that
> reason is that interrupt for receiving packets is targeted to one processor.
> Also was discussed that processor afinity for DPC is possible to force to
> some processor.
> My question is if such work has sense to avoid such work if it is crazy idea.
> Has sense to try to force some DPC routines to other processor than is
> interrupted by receiving packets ?
> Or system does load balancing the best how it can be and my forcing of DPC
> routines to other processor has not sense in my driver ?
>
> Thx!
> Peter

Thank you Stefan for interesting links, it contains much of info which I need
now.
I have one additional question to problem relating to assymetric overloaded
processors:
By info which I have, which processor os targeted in interrupts for
receiving packets is depending on APIC, maybe settings in BIOS.
I have W2K3 machine configured like router. When about 620Mbit flow one
processor is about 90% #without# my driver installed, so this is tight place
in system, not in my driver.
Has sense to try for example manually to set other interrupt in BIOS ? Or
some other settings ?

Thx !
Peter


"Stephan Wolf [MVP]" wrote:

> If one CPU is 100% busy handling interrupts, then what you need is to
> find some way to implement "interrupt moderation". That is, you should
> not have a one-to-one relation between received/sent frames and
> interrupts. Most modern network cards/chips allow for interrupt
> moderation by e.g. delaying interrupts to some extent and/or counting
> frames before issuing an interrupt.
>
> Run perfmon to visualize the number of interrupts and DPCs.
>
> See also:
>
> "Analyze Driver Performance"
> http://www.microsoft.com/whdc/driver...m/drvperf.mspx
>
> "High Performance Network Adapters and Drivers"
> http://www.microsoft.com/whdc/device...ters-Drvs.mspx
>
> "Windows Scalable Networking Initiative"
> http://www.microsoft.com/whdc/device/network/scale.mspx
>
> Stephan
> ---
> Peter wrote:
> > When two processor machine with W2K3 is overloaded by high network traffic,
> > one processor raises almost to 100% and second only 20-30%.
> > In some mails in this group this problem was discussed and explained that
> > reason is that interrupt for receiving packets is targeted to one processor.
> > Also was discussed that processor afinity for DPC is possible to force to
> > some processor.
> > My question is if such work has sense to avoid such work if it is crazy idea.
> > Has sense to try to force some DPC routines to other processor than is
> > interrupted by receiving packets ?
> > Or system does load balancing the best how it can be and my forcing of DPC
> > routines to other processor has not sense in my driver ?
> >
> > Thx!
> > Peter
>
>

Peter, when you say #without# your driver do you mean with a different
network driver?
As Stefan says it will most likely come down to interrupt moderation.

You may already know this, but for most network drivers if go to the
device properties in device manager, and click the advanced tab...
depending on the driver you can normally modify the driver to be for
either high throughput, or low latency. In your case with 620Mbit you
will need high throughput. On my Marvell Yukon I can also set things
like the maximum number of IRQs per second.

Peter,

> When two processor machine with W2K3 is overloaded by high network traffic,
> one processor raises almost to 100% and second only 20-30%.
> In some mails in this group this problem was discussed and explained that
> reason is that interrupt for receiving packets is targeted to one processor.


Network card interrupt can hardly qualify for being either fixed or SMI
or NMI or EXTINT or start-up, does it???? Therefore, the OS,
apparently, programs it to be distributed dynamically. When interrupts
are distributed dynamically, they are dispensed to the least busy CPU
via bus arbitration protocol, which is completely transparent to the
software (please check Intel documentation - namely, IOAPIC Manual and
Developers' Manual, Volume 3).


> Also was discussed that processor afinity for DPC is possible to force to
> some processor.

Indeed, this is what KeSetTargetProcessorDpc() is for

> Or system does load balancing ......

Unless you have specified some particular CPU for DPC by calling
KeSetTargetProcessorDpc(), it will run on the CPU that inserts DPC into
a queue - there is no load-balancing here. However, DPCs are normally
queued by ISRs, and, as I have already said, interrupts are dispensed
to the least busy CPU. Therefore, we can speak about indirect
load-balancing......

> Has sense to try to force some DPC routines to other processor than is
> interrupted by receiving packets ?

It is hard to say anything without knowing the exact situation, but I
would suggest to leave it to the system to decide how to do things.


Anton Bassov



Peter wrote:
> When two processor machine with W2K3 is overloaded by high network traffic,
> one processor raises almost to 100% and second only 20-30%.
> In some mails in this group this problem was discussed and explained that
> reason is that interrupt for receiving packets is targeted to one processor.
> Also was discussed that processor afinity for DPC is possible to force to
> some processor.
> My question is if such work has sense to avoid such work if it is crazy idea.
> Has sense to try to force some DPC routines to other processor than is
> interrupted by receiving packets ?
> Or system does load balancing the best how it can be and my forcing of DPC
> routines to other processor has not sense in my driver ?
>
> Thx!
> Peter

See also

Microsoft Knowledge Base Article 892100
"The processor load is not distributed across multiple processors on a
computer that is running Windows Server 2003, Windows 2000 Server, or
Windows NT 4.0"
http://support.microsoft.com/kb/892100

It explains how to set the ProcessorAffinityMask registry entry for
NDIS.

Stephan

It's time to read IOAPIC Manual... :-)
but other q:
when using KeSetTargetProcessorDpc()
it it should be fine to know what processor is overloaded and what not.
How to retrieve this info in kernel mode ? (if it is possible)

Peter


"" wrote:

> Peter,
>
> > When two processor machine with W2K3 is overloaded by high network traffic,
> > one processor raises almost to 100% and second only 20-30%.
> > In some mails in this group this problem was discussed and explained that
> > reason is that interrupt for receiving packets is targeted to one processor.
>
>
> Network card interrupt can hardly qualify for being either fixed or SMI
> or NMI or EXTINT or start-up, does it???? Therefore, the OS,
> apparently, programs it to be distributed dynamically. When interrupts
> are distributed dynamically, they are dispensed to the least busy CPU
> via bus arbitration protocol, which is completely transparent to the
> software (please check Intel documentation - namely, IOAPIC Manual and
> Developers' Manual, Volume 3).
>
>
> > Also was discussed that processor afinity for DPC is possible to force to
> > some processor.
>
> Indeed, this is what KeSetTargetProcessorDpc() is for
>
> > Or system does load balancing ......
>
> Unless you have specified some particular CPU for DPC by calling
> KeSetTargetProcessorDpc(), it will run on the CPU that inserts DPC into
> a queue - there is no load-balancing here. However, DPCs are normally
> queued by ISRs, and, as I have already said, interrupts are dispensed
> to the least busy CPU. Therefore, we can speak about indirect
> load-balancing......
>
> > Has sense to try to force some DPC routines to other processor than is
> > interrupted by receiving packets ?
>
> It is hard to say anything without knowing the exact situation, but I
> would suggest to leave it to the system to decide how to do things.
>
>
> Anton Bassov
>
>
>
> Peter wrote:
> > When two processor machine with W2K3 is overloaded by high network traffic,
> > one processor raises almost to 100% and second only 20-30%.
> > In some mails in this group this problem was discussed and explained that
> > reason is that interrupt for receiving packets is targeted to one processor.
> > Also was discussed that processor afinity for DPC is possible to force to
> > some processor.
> > My question is if such work has sense to avoid such work if it is crazy idea.
> > Has sense to try to force some DPC routines to other processor than is
> > interrupted by receiving packets ?
> > Or system does load balancing the best how it can be and my forcing of DPC
> > routines to other processor has not sense in my driver ?
> >
> > Thx!
> > Peter
>
>

Peter,

> it it should be fine to know what processor is overloaded and what not.
> How to retrieve this info in kernel mode ? (if it is possible)


Again, I have to repeat myself - APIC arbitration protocol is
completely transparent to *ANY* software, including the OS itself.
Therefore, there is absolutely no way to retrieve this info. The only
thing you could do is to obtain some obsolete info from APIC/IOAPIC
registers. Why is this info obsolete??? Simply because the state of
APIC bus is just too volatile thing - any info that software is able to
get on the subject is already obsolete. APIC and IOAPIC hardware is the
only one who can get real-time info on the subject


Anton Bassov


Peter wrote:
> It's time to read IOAPIC Manual... :-)
> but other q:
> when using KeSetTargetProcessorDpc()
> it it should be fine to know what processor is overloaded and what not.
> How to retrieve this info in kernel mode ? (if it is possible)
>
> Peter
>
>
> "" wrote:
>
> > Peter,
> >
> > > When two processor machine with W2K3 is overloaded by high network traffic,
> > > one processor raises almost to 100% and second only 20-30%.
> > > In some mails in this group this problem was discussed and explained that
> > > reason is that interrupt for receiving packets is targeted to one processor.
> >
> >
> > Network card interrupt can hardly qualify for being either fixed or SMI
> > or NMI or EXTINT or start-up, does it???? Therefore, the OS,
> > apparently, programs it to be distributed dynamically. When interrupts
> > are distributed dynamically, they are dispensed to the least busy CPU
> > via bus arbitration protocol, which is completely transparent to the
> > software (please check Intel documentation - namely, IOAPIC Manual and
> > Developers' Manual, Volume 3).
> >
> >
> > > Also was discussed that processor afinity for DPC is possible to force to
> > > some processor.
> >
> > Indeed, this is what KeSetTargetProcessorDpc() is for
> >
> > > Or system does load balancing ......
> >
> > Unless you have specified some particular CPU for DPC by calling
> > KeSetTargetProcessorDpc(), it will run on the CPU that inserts DPC into
> > a queue - there is no load-balancing here. However, DPCs are normally
> > queued by ISRs, and, as I have already said, interrupts are dispensed
> > to the least busy CPU. Therefore, we can speak about indirect
> > load-balancing......
> >
> > > Has sense to try to force some DPC routines to other processor than is
> > > interrupted by receiving packets ?
> >
> > It is hard to say anything without knowing the exact situation, but I
> > would suggest to leave it to the system to decide how to do things.
> >
> >
> > Anton Bassov
> >
> >
> >
> > Peter wrote:
> > > When two processor machine with W2K3 is overloaded by high network traffic,
> > > one processor raises almost to 100% and second only 20-30%.
> > > In some mails in this group this problem was discussed and explained that
> > > reason is that interrupt for receiving packets is targeted to one processor.
> > > Also was discussed that processor afinity for DPC is possible to force to
> > > some processor.
> > > My question is if such work has sense to avoid such work if it is crazy idea.
> > > Has sense to try to force some DPC routines to other processor than is
> > > interrupted by receiving packets ?
> > > Or system does load balancing the best how it can be and my forcing of DPC
> > > routines to other processor has not sense in my driver ?
> > >
> > > Thx!
> > > Peter
> >
> >

I ment info like for example is possible to retrieve with help of performance
monitor functions in user mode, or maybe in kernel mode is possible to
retrieve info about DPC queue length. By such info I can to know what
processor is good target for
KeSetTargetProcessorDpc()
I think using of this function can to help, because all my software
interrupts I can to force to some processor. But how to retrieve suitable
processor number in kernel mode ?

Peter



"" wrote:

>
> Peter,
>
> > it it should be fine to know what processor is overloaded and what not.
> > How to retrieve this info in kernel mode ? (if it is possible)
>
>
> Again, I have to repeat myself - APIC arbitration protocol is
> completely transparent to *ANY* software, including the OS itself.
> Therefore, there is absolutely no way to retrieve this info. The only
> thing you could do is to obtain some obsolete info from APIC/IOAPIC
> registers. Why is this info obsolete??? Simply because the state of
> APIC bus is just too volatile thing - any info that software is able to
> get on the subject is already obsolete. APIC and IOAPIC hardware is the
> only one who can get real-time info on the subject
>
>
> Anton Bassov
>
>
> Peter wrote:
> > It's time to read IOAPIC Manual... :-)
> > but other q:
> > when using KeSetTargetProcessorDpc()
> > it it should be fine to know what processor is overloaded and what not.
> > How to retrieve this info in kernel mode ? (if it is possible)
> >
> > Peter
> >
> >
> > "" wrote:
> >
> > > Peter,
> > >
> > > > When two processor machine with W2K3 is overloaded by high network traffic,
> > > > one processor raises almost to 100% and second only 20-30%.
> > > > In some mails in this group this problem was discussed and explained that
> > > > reason is that interrupt for receiving packets is targeted to one processor.
> > >
> > >
> > > Network card interrupt can hardly qualify for being either fixed or SMI
> > > or NMI or EXTINT or start-up, does it???? Therefore, the OS,
> > > apparently, programs it to be distributed dynamically. When interrupts
> > > are distributed dynamically, they are dispensed to the least busy CPU
> > > via bus arbitration protocol, which is completely transparent to the
> > > software (please check Intel documentation - namely, IOAPIC Manual and
> > > Developers' Manual, Volume 3).
> > >
> > >
> > > > Also was discussed that processor afinity for DPC is possible to force to
> > > > some processor.
> > >
> > > Indeed, this is what KeSetTargetProcessorDpc() is for
> > >
> > > > Or system does load balancing ......
> > >
> > > Unless you have specified some particular CPU for DPC by calling
> > > KeSetTargetProcessorDpc(), it will run on the CPU that inserts DPC into
> > > a queue - there is no load-balancing here. However, DPCs are normally
> > > queued by ISRs, and, as I have already said, interrupts are dispensed
> > > to the least busy CPU. Therefore, we can speak about indirect
> > > load-balancing......
> > >
> > > > Has sense to try to force some DPC routines to other processor than is
> > > > interrupted by receiving packets ?
> > >
> > > It is hard to say anything without knowing the exact situation, but I
> > > would suggest to leave it to the system to decide how to do things.
> > >
> > >
> > > Anton Bassov
> > >
> > >
> > >
> > > Peter wrote:
> > > > When two processor machine with W2K3 is overloaded by high network traffic,
> > > > one processor raises almost to 100% and second only 20-30%.
> > > > In some mails in this group this problem was discussed and explained that
> > > > reason is that interrupt for receiving packets is targeted to one processor.
> > > > Also was discussed that processor afinity for DPC is possible to force to
> > > > some processor.
> > > > My question is if such work has sense to avoid such work if it is crazy idea.
> > > > Has sense to try to force some DPC routines to other processor than is
> > > > interrupted by receiving packets ?
> > > > Or system does load balancing the best how it can be and my forcing of DPC
> > > > routines to other processor has not sense in my driver ?
> > > >
> > > > Thx!
> > > > Peter
> > >
> > >
>
>

In Perfmon you can record DPCs Queued/Sec if that helps. I'm not sure
how can get hold that figure in a kernel driver to decided which cpu to
set KeSetTargetProcessorDpc. To be honest if you're getting 100% cpu
usage on network packets something more fundamental might be going
wrong. What user mode application is reading/creating these packets?