Disk latency generate High CPU

Discussion in 'Windows Server' started by NeerinckB, Feb 14, 2008.

  1. NeerinckB

    NeerinckB Guest

    Hi,

    I suspect Windows Server to indicate in Task Manager High CPU Utilization
    when in fact there are Disk I/O Latency and not heavy CPU usage.

    I would like to have feedback from other professional about this.

    I'm currently working with hundred of Windows Server that are virtualized
    using VMware, and we get often complain about Windows Server VM that have
    High CPU Usage that we can't explain while we suspect the storage to be
    responding slowly.

    Thanks,
     
    NeerinckB, Feb 14, 2008
    #1
    1. Advertisements

  2. I would expect it the other way around

    if you have high IO latency you would see lower CPU utilization "because
    your application is *slowed down* by not able to quickly do IO's"

    often I actually have seen this on server applications where a disk
    bottleneck was solved and a CPU bottleneck appeared, as suddenly the
    application was doing fast IO's, and now was *hammering* the CPU with more
    "transactions"
     
    Edwin vMierlo [MVP], Feb 14, 2008
    #2
    1. Advertisements

  3. NeerinckB

    Mark Perry Guest

    Hi I can confirm that this is an issue, and also have figured out a way around this.

    The High CPU latency seems related to poor disk IO as it is wasting cycles waiting for data. If you run up perfmon and check the stats you will probably see under heavy utilisation the disk I/O avg disk que flatlining at the top of the graph as the ques go off the chart. The CPU and memory paging also follow suit.

    Doing some performance testing I was able to see that I could not get a sustanined write to disk over 10MB/s when the underlying OS could write to disk at near 300MB/s. Web portal boxes tend to be fine like this but any Server running database apps e.g. Exchange, SQL tend to be horrific under load.

    I have tested under both Server and ESX and experienced the same issue. Therefore I concluded that the virtual preallocated vmdk disk was the cause as on Linux it resembles a network block device v similar to the Linux loopback device. I then created a loopback device to test performance which twice as fast as a vmdk block device, I also tested VirtualBox which gave similar results as the loopback so looseley concluded that it was using the same technology. Not sure why vmwares version of a loopback device is half speed of Linux own loopback device though???

    Since these were still horrific speeds I decided to migrate my existing VM's to raw physical disks, ESX does not support this and if you look at ESX speed test white papers ESX iSCSI and Fibre Channel Sppeds are very good, not least becasue these disks are raw.... Therefore I built a shiny new server, the only problem I encountered was Vmware raw disk suppport does not include Linux LVM support (probably not an issue if your running M$ host on basic disks not sure about dynamic or a flat ext partition, don't use XFS if you want snapshotting it hangs), so Vmware wouldn't see my raw partitions on LVM no problem just get the vmgbd (vmware generic block device) patch and create soft links from /dev/sdx -> /dev/VG-Vol-Group/LV-Logical-Volume and point to those. While setting up use the root account as you spend wasteful time getting access to the partitions.

    Once I had raw partitoins set up I just created loopback devices with my preallocated (only works with preallocated disks) using losetup command and then dd to dump the data from the vmdk to raw disk. The raw disks must be the same size or bigger that the original disk unless you really want to mess about. In the case of larger physical disk you can boot your os, make sure it works then shutdown and run ntfsresize against it to expand the ntfs partition to fill the partition.

    Once done you now have near native disk speeds.

    Sorry if this isn't clear but I went through a lot of pain to get this far...
     
    Mark Perry, Jun 16, 2008
    #3
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.