csrss.exe High I/O in combination with Terminal service

Discussion in 'Windows Server' started by Vincent, Oct 17, 2008.

  1. Vincent

    Vincent Guest

    Hi,

    Currently I’m running into some issues which occur on each and every
    windows 2003 server that I’m managing:

    When someone logs in on a server the csrss.exe process starts to use up a
    lot of disk I/O. It seems to have a minimum of 10 MB/s with a maximum of 80
    MB/s although the average seems to be around 30 MB/s. The high I/O continues
    until the user logs off. The high I/O also stops completely when (and this is
    especially weird) the Terminal service client screen is minimized, but high
    disk activity will resume as soon as the screen is restored.

    I am aware of knowledge base article 934330, and even though the article in
    question states the fix is foor the issue that crss.exe utilizes a lot of
    cpu instead of high I/O, I decided to install the hotfix on a server to see
    if the problems are related. Unfortunately it seems they’re not.

    As I already said I'm experiencing this issue on multiple windows 2k3
    server versions (web, standard, standard R2). They are all running service
    pack 2 and are fully patched. The hardware differs as well. I've seen this
    problem occur on Dell and HP servers so it is practically impossible that its
    hardware related.

    I'm at loss as to what the cause is. Does anybody have a clue?

    Thanks in advance,
    Vincent
     
    Vincent, Oct 17, 2008
    #1
    1. Advertisements

  2. I installed the 934330 hotfix received from MS Support (see
    http://support.microsoft.com/kb/934330 that explains a similar problem
    with csrss.exe) but it didn't solved the problem.

    -- rpr.
     
    Robert Premuž, Nov 3, 2008
    #2
    1. Advertisements

  3. Vincent

    jolteroli Guest

    Do the session, the "freaking csrss" belongs to, also have a process named
    "ntvdm" running?

    -jolt
     
    jolteroli, Nov 3, 2008
    #3
  4. Vincent

    jolteroli Guest

    Not sure if this will help...

    If you look in proc-explorer on the threads-tab in the csrss process, you
    should find a thread causing all this IO and having a pretty high
    cpu/context-switch-delta value.

    If you suspend this thread does the IO drop? (Do this on the csrss of a
    test-user session)

    What is the "starting address" and the call [Stack] of the thread?

    -jolt

    There is no ntvdm process running on the server.

    -- rpr.
     
    jolteroli, Nov 4, 2008
    #4
  5. Vincent

    jolteroli Guest

    Hey Robert,

    I don't want bother you, but you've not configured symbols. If you want to
    have them:

    1.) Install latest windbg.
    2.) Copy latest dbghelp.dll (in the windbg directory) to system32
    3.) Create a directory for symbol files, e.g.: f:\symbols
    4.) Set system-wide environment variable:
    _NT_SYMBOL_PATH=SRV*f:\symbols*http://msdl.microsoft.com/download/symbols
    5.) Download symbols for a single image by:
    symchk c:\winnt\system32\winsrv.dll
    symchk c:\winnt\system32\csrss.exe
    symchk c:\winnt\system32\win32k.sys
    symchk c:\winnt\system32\ntkrnpa.exe
    symchk c:\winnt\system32\ntdll.dll
    ... and so on...
    Or download any image in system32 recursively:
    symchk /r c:\winnt\system32\*
    6.) In the Process Explorer, configure the symbol path, to f:\symbols

    You can put
    dbgeng.dll
    dbghelp.dll
    SymbolCheck.dll
    symchk.exe
    symsrv.dll
    on a USB pen. So you don't need the Debugging Tools any time.

    If you now view the threads and their stack trace, it might look like:

    winsrv.dll!StartCreateSystemThreads:
    ntkrnlpa.exe!KiSwapContext+0x2f
    ntkrnlpa.exe!KiSwapThread+0x8a
    ntkrnlpa.exe!KeWaitForMultipleObjects+0x284
    win32k.sys!RawInputThread+0x4f3
    win32k.sys!xxxCreateSystemThreads+0x60
    win32k.sys!NtUserCallOneParam+0x23
    ntkrnlpa.exe!KiFastCallEntry+0xfc
    ntdll.dll!KiFastSystemCallRet
    winsrv.dll!NtUserCallOneParam+0xc

    And you can see which functions are involved in the freaking thread. I would
    start and view the stack traces

    Once, when the RDP window is minimized has no IO
    Once, when the RDP window is maximized and produce IO

    Any difference?

    -jolt
     
    jolteroli, Nov 9, 2008
    #5
  6. Vincent

    jolteroli Guest

    Hee Rob,
    Real men don't click, cool... :)
    Since our servers (not having this issue) doesn't fork to KiDeliverApc, I
    guess there is constantly arriving keyb data (that is what RawInputThread
    RIT "polls" for) and that's causing the IO. I have no clue what is causing
    the keyb data...

    In german we have the Systemcontrol "Eingabehilfen" with the symbol of Steve
    Balmers wheelchair. Check if any enhanced services is turned on there...

    ATI's stupid HotKeyPolling service comes to mind too...

    May be something in this direction. For me it seems the data comes from the
    keyb/driver.
    Any additional help would need a debugger, so it was a waste of time. Sorry
    Rob.

    -jolt
     
    jolteroli, Nov 13, 2008
    #6
  7. Vincent

    TP Guest

    Robert and Vincent,

    What are the exact problem symptoms you are experiencing? Is
    the server running slow? Are users complaining? If yes, what
    lead you to the conclusion that this particular reading is the
    source of the problems you are having?

    I have looked at the numbers you are seeing and this must be
    a bug. I see this on every 2003 server I have examined so far
    (since reading your post). If these numbers were true the
    servers would grind to a halt.

    Thanks.

    -TP
     
    TP, Nov 13, 2008
    #7
  8. Vincent

    MrM Mills Guest

    Found your post while trying to understand Microsoft Office Server 2007 consumption of resources (MOSS 2007\SharePoint).

    My SharePoint Central Administration Server is a W2k3 R2 SP2 Std Ed X64 server with 8GB of memory.
    My IO Delta climbs at times from 64KB to 84GB (not a typo ...84GB!)this is for the CRSS.exe (Client Server Runtime Process) when I view it in the latest version of Process Explorer.

    My "I\O Other Bytes" is reading 19 Trillion+, equivelant to 19 Tereabytes????. The server itself is on a 30GB and 40GB partitions so the 19Trillion Terabytes may be I\O's from reading the SQL DB's its attached to which reside on another server. But the SQL db's its talking to total less than 200GB?

    dumb founded....this server is showing very few eventlog errors and was only rebooted 12 hours ago.

    In Process Explorer I'm also seeing a lot (as in hundreds) of "Contentions" for different instancesof the MS Seach Component mssdmn.exe

    MSSearch.exe is also racking up a significant amount or PageFaults??

    Will be openign a case with MS as soon as schedule permits.
     
    MrM Mills, Nov 19, 2008
    #8
  9. Vincent

    kC C Guest

    hi Guys just a quick question to you all suffering this issue, wheres your Page filing pointing to and whats it set to?

     
    kC C, Apr 6, 2011
    #9
  10. Vincent

    David Ludvik Guest

    Hi Guys,
    this hot-fix should fix the issue
    http://support.microsoft.com/kb/956438

     
    David Ludvik, May 4, 2011
    #10
  11. Vincent

    David Ludvik Guest

    Hi Guys,
    This should fix the issue.

    http://support.microsoft.com/kb/956438
     
    David Ludvik, May 4, 2011
    #11
  12. Vincent

    Guest Guest

    Guest, Jul 24, 2013
    #12
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.