DMA transfer speed issues on PCI card with PCI and PCI-X busses

Discussion in 'Windows Vista Drivers' started by Oliver, Nov 25, 2006.

  1. Oliver

    Oliver Guest

    Hi all,

    we're using a PCI card with large on-board memory and a PCI interface chip with DMA busmaster engine
    (PCI9656 from PLX but only using 32 bit width). Data transfer is done using Scatter Gather DMA and
    is running stable. The card is working in PCI and PCI-X busses with 33 MHz and 66 MHz.

    When measuring the DMA transfer speed I get some strange results:

    used in a standard 32 bit PCI slot:
    read: card to PC memory: 80-100 MB/s
    write: PC memory to card: 80-100 MB/s

    used in a 64 bit PCI-X slot with 66 MHz:
    read: card to PC memory: 200-220 MB/s
    write: PC memory to card: 40-50 MB/s

    using an older design with a pure 33 MHz 32 bit interface chip (PCI9080)in a PCI-X slot:
    read: card to PC memory: 80-100 MB/s
    write: PC memory to card: 40-50 MB/s

    Concerning the read direction the speed is as expected but as soon as a PCI-X slot is involved the
    bus transfer speed in write direction drops a lot. The Scatter-Gather list is located in PC memory
    what should make access easier in write direction as the PCI bus need not to change direction.

    We tested at least 20 different PCI-X systems and get these results with ±10%. It can't be a local
    bus issue as the card is at least able to stream data with 100 MB/s on a 32 bit PCI slot.

    Anybody an idea where to start searching and which setup to change?
    When monitoring PCI bus signal I see that in write direction the transfer drops and is restarted
    after only a few bytes of transfer. But why? We modified the design to be sure that the local bus is
    able to run with even 250 MB/s with no latency.
    There're no other card accesses while DMA is running, the driver simply waits for the interrupt.

    Any help is appreciated.

    Best regards
    Oliver, Nov 25, 2006
    1. Advertisements

  2. You can't use PCI-X _bus_ with PCI9656. PCI9656 is PCI-only device.
    What you use is PCI-X _slot_ in PCI mode.
    First you seem confused about the naming of PCI transfers.
    For bus-master cards:
    PCI-to-memory = Write
    Memory-to-PCI = Read

    Second, in PLX IO accelerators you could program the PCI command used
    in DMA transactions (See register 6Ch). The default For Memory-to-PCI
    DMA is "memory read line". Some chipsets, e.g. AMD 8132 interpret
    "memory read line" command as "read one processor cache line" i.e. in
    case of Opteron 64 bytes. Reading just 64 bytes leads to suboptimal
    Program DMA to use "memory read multiple" command and on many systems
    you will see immediate increase in performance.

    Third, you didn't tell us about your local bus. May be the bottleneck
    is on the local side of the bridge?

    Fourth, your question doesn't really belong to m.p.d.d.d. You are just
    lucky that I am bored.
    already5chosen, Nov 25, 2006
    1. Advertisements

  3. Oliver

    Oliver Guest

    You're right, that was my stupid driver viewpoint.
    Many thanks for that hint. It bursted the transfer to 180-200 MB/s for
    PC to card transfers
    Yes, I was really lucky - where do you think my question would have been
    correctly placed?
    Oliver, Nov 27, 2006
  4. Well, many of us are also concerned here. Maybe try in "comp.arch".

    Stephan Wolf [MVP], Nov 27, 2006
  5. Agreed. You will see all kinds of different "bus throughput" numbers
    between various machines and chipsets. You will sometimes even see
    different numbers in slot X vs. Slot Y of the same machine. Simply
    because some slots are behind a bridge while others are not.

    If you want to actually analyze the PCI bus, get yourself a bus
    analyzer like VMETRO. These are not quite cheap, however. But you will
    usually see many weird effects and command sequences on the bus, which
    you would not expect to see. Then optimize your hardware and driver to
    better organize bus commands etc.

    Stephan Wolf [MVP], Nov 27, 2006
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.