SP2 NT Backup Hangs and leaves system in unresponsive state

Discussion in 'Windows Small Business Server' started by Tom Schwartz, Jan 18, 2008.

  1. Tom Schwartz

    Tom Schwartz Guest

    Dell Server with 6 total drives 2 in hardware Raid 0 for the system and 4 in
    Hardware Raid 5 for data. NT Backup configured for nightly full system
    backup of drive C: and Drive D: Exchange Stores and System state to
    separately attached USB 2.0 External harddrive. This setup works fine for up
    to 14 days but randomly during NT Backup the sytem will become unresponsive
    after completing the backup of the two drives C: and D: prior to doing the
    Exchange Stores or system state

    Application Event Logs Show
    Backup Starting on C:
    Backup Completing C: Drive
    Backup Starting on D:
    Backup Completing D: Drive
    -----
    Nothing else until system is rebooted

    Backup Log File only shows

    1/17/2008 8:00 PM
    -------------------------------
    Date: 1/17/2008
    Time: 8:00 PM
    User: SYSTEM
    -------------------------------

    Backup Runner started.
    Launching NTBackup: ntbackup.exe backup "@C:\Program Files\Microsoft Windows
    Small Business Server\Backup\Small Business Backup Script.bks" /d "SBS Backup
    created on 1/17/2008 at 8:00 PM" /v:yes /r:no /rs:no /hc:eek:ff /m normal /j
    "Small Business Server Backup Job" /l:s /f "\\Serverroom\Backup Network
    Files\Backup Files\Small Business Server Backup (02).bkf" /UM

    Strange that the file does not log the completion of either drive ???

    Normal Complete File


    1/16/2008 8:00 PM
    -------------------------------
    Date: 1/16/2008
    Time: 8:00 PM
    User: SYSTEM
    -------------------------------

    Backup Runner started.
    Launching NTBackup: ntbackup.exe backup "@C:\Program Files\Microsoft Windows
    Small Business Server\Backup\Small Business Backup Script.bks" /d "SBS Backup
    created on 1/16/2008 at 8:00 PM" /v:yes /r:no /rs:no /hc:eek:ff /m normal /j
    "Small Business Server Backup Job" /l:s /f "\\Serverroom\Backup Network
    Files\Backup Files\Small Business Server Backup (01).bkf" /UM
    NTBACKUP LOG FILE: C:\Documents and Settings\SBS Backup User\Local
    Settings\Application Data\Microsoft\Windows NT\NTBackup\data\backup02.log
    =====================<BEGIN NTBACKUP LOG FILE>=====================
    Backup Status
    Operation: Backup
    Active backup destination: File
    Media name: "Small Business Server Backup (01).bkf created 1/16/2008 at 8:00
    PM"

    Volume shadow copy creation: Attempt 1.
    Backup (via shadow copy) of "C: SBS 2003 OS"
    Backup set #1 on media #1
    Backup description: "SBS Backup created on 1/16/2008 at 8:00 PM"
    Media name: "Small Business Server Backup (01).bkf created 1/16/2008 at 8:00
    PM"

    Backup Type: Normal

    Backup started on 1/16/2008 at 8:02 PM.
    Backup completed on 1/16/2008 at 8:47 PM.
    Directories: 3643
    Files: 36571
    Bytes: 9,762,922,825
    Time: 45 minutes and 26 seconds
    Backup (via shadow copy) of "D: Holy Family Data"
    Backup set #2 on media #1
    Backup description: "SBS Backup created on 1/16/2008 at 8:00 PM"
    Media name: "Small Business Server Backup (01).bkf created 1/16/2008 at 8:00
    PM"

    Backup Type: Normal

    Backup started on 1/16/2008 at 8:47 PM.
    Backup completed on 1/16/2008 at 9:56 PM.
    Directories: 3193
    Files: 57154
    Bytes: 19,749,484,955
    Time: 1 hour, 8 minutes, and 49 seconds
    Backup of "MAINHFCC\Microsoft Information Store\First Storage Group"
    Backup set #3 on media #1
    Backup description: "SBS Backup created on 1/16/2008 at 8:00 PM"
    Media name: "Small Business Server Backup (01).bkf created 1/16/2008 at 8:00
    PM"

    Backup Type: Normal

    Backup started on 1/16/2008 at 9:56 PM.
    Backup completed on 1/16/2008 at 10:01 PM.
    Directories: 4
    Files: 5
    Bytes: 1,951,441,486
    Time: 5 minutes and 13 seconds
    Backup (via shadow copy) of "System State"
    Backup set #4 on media #1
    Backup description: "SBS Backup created on 1/16/2008 at 8:00 PM"
    Media name: "Small Business Server Backup (01).bkf created 1/16/2008 at 8:00
    PM"

    Backup Type: Copy

    Backup started on 1/16/2008 at 10:01 PM.
    Backup completed on 1/16/2008 at 10:05 PM.
    Directories: 254
    Files: 2822
    Bytes: 616,140,927
    Time: 4 minutes and 2 seconds

    ----------------------

    Verify Status
    Operation: Verify After Backup
    Active backup destination: File
    Active backup destination: \\Serverroom\Backup Network Files\Backup
    Files\Small Business Server Backup (01).bkf

    Verify of "C:"
    Backup set #1 on media #1
    Backup description: "SBS Backup created on 1/16/2008 at 8:00 PM"
    Verify started on 1/16/2008 at 10:06 PM.
    Verify completed on 1/16/2008 at 10:26 PM.
    Directories: 3643
    Files: 36571
    Different: 0
    Bytes: 9,762,922,825
    Time: 20 minutes and 3 seconds

    Verify of "D:"
    Backup set #2 on media #1
    Backup description: "SBS Backup created on 1/16/2008 at 8:00 PM"
    Verify started on 1/16/2008 at 10:26 PM.
    Verify completed on 1/16/2008 at 11:06 PM.
    Directories: 3193
    Files: 57154
    Different: 0
    Bytes: 19,749,484,109
    Time: 40 minutes and 6 seconds

    Verify of "MAINHFCC\Microsoft Information Store\First Storage Group"
    Backup set #3 on media #1
    Backup description: "SBS Backup created on 1/16/2008 at 8:00 PM"
    Verify started on 1/16/2008 at 11:06 PM.
    Verify completed on 1/16/2008 at 11:10 PM.
    Directories: 4
    Files: 0
    Different: 0
    Bytes: 1,951,441,486
    Time: 3 minutes and 52 seconds

    Verify of "System State"
    Backup set #4 on media #1
    Backup description: "SBS Backup created on 1/16/2008 at 8:00 PM"
    Verify started on 1/16/2008 at 11:10 PM.
    Verify completed on 1/16/2008 at 11:11 PM.
    Directories: 254
    Files: 2822
    Different: 0
    Bytes: 616,140,927
    Time: 1 minute and 13 seconds

    ----------------------

    =======================<END NTBACKUP LOG FILE>=====================
    NTBackup finished the backup with no errors.
    Backup ended at Wednesday, January 16, 2008 11:11 PM
    Backup Runner finished.


    Any Troubleshooting Help Appreciated.....
     
    Tom Schwartz, Jan 18, 2008
    #1
    1. Advertisements

  2. What makes you think its NT backup causing the problem?

    From the log it seems it finished OK. Is there anything else scheduled to
    run around that time?

    Marcus
     
    Marcus Robinson, Jan 18, 2008
    #2
    1. Advertisements

  3. <snip>

    Hmmm. As the other reply says, your backup job itself looks fine from this
    log. However, I'm puzzled: you say you're using a USB hard drive, but your
    backup job is going across the network: "\\Serverroom\Backup Network
    Files\Backup Files\Small Business Server Backup (02).bkf"

    FWIW, I have never had good luck backing up to a NAS or other network
    destination using NTBackup. I'd suggest a locally attached USB hard drive
    (multiple ones, actually, so you can swap them out). Or, invest in a good
    tape drive. Or, get third party backup software like BackupExec.

    Also, you haven't said what "unresponsive" actually means...is your server
    frozen solid?
     
    Lanwench [MVP - Exchange], Jan 18, 2008
    #3
  4. Tom Schwartz

    Holz Guest

    How much empty space on each drive?
    where is the temp directory located?
    Size of the swap file?
    How much total memory on the system?
     
    Holz, Jan 18, 2008
    #4
  5. Tom Schwartz

    Tom Schwartz Guest

    Marcus - Thanks for the look.

    I know the process fails because the when it works the sytem is not
    unresponsive, the SBS Backup Log is complete with all 4 items to backup, ie
    Drive C, D, Exchange, System State and verifies are done on all four.

    When it hangs, the system log completely shuts down, nothing else is
    attempted or logged either by the sytem or by NT Backup as per the first
    message. Pressing a key on the keyboard causes the display to come out of
    hibernation but nothing else happens until to hit the hard reset.

    Tom
     
    Tom Schwartz, Jan 18, 2008
    #5
  6. I missed the split between the two logs in your first message!

    Only thing I have seen similar (i.e. happening randomly) has been due to
    issues with VSS. The latest update for VSS has helped on occasions (either
    of http://support.microsoft.com or a hotfix from MS). May not be the case
    for you though...

    Marcus
     
    Marcus Robinson, Jan 18, 2008
    #6
  7. Tom Schwartz

    Tom Schwartz Guest

    System Drive is 20Gig with 7 Gig Available
    Data Drive is 67 Gig with 52Gig Available
    2 Gig Memory
    Swap file is on C: at 2 Gig

    The detailed log is an example of when everything works correctly and yes I
    have been playing with the Backup Destination as I read that there were some
    folks having issues with USB drives attached to SBS Server. I tried the
    Network drive and work for a while and then failed. Just trying to
    troubleshoot whether it is a Backup Destination issue or something else.

    Another piece of the puzzle. If I don't run ISINTEG and fix the errors
    after the HARD REBOOT, the system backup will continue to fail not everytime
    but on most runs and look the system up solid. After running ISINTEG and
    fixing a few errors, the backup works for up to 14 days or so before crashing
    again. I have never had a problem with ISINTEG finding errors when
    DISMOUNTING the stores BEFORE a crash occurs. I have only found them AFTER
    a crash.
     
    Tom Schwartz, Jan 18, 2008
    #7
  8. Tom Schwartz

    Holz Guest

    What happens when you backup locally, say to your D Drive?
     
    Holz, Jan 18, 2008
    #8
  9. Tom

    Make sure you have the latest NIC drivers from DELL compatible with SP2 and
    that you have installed the post SP2 patches from Microsoft Update.

    Next after resolving this issue and insuring that you have at least 2
    verified total backups I recommend you reconfigure the server to get off of
    Raid 0 and use Raid 1 for your OS. Raid 0 is a VERY bad choice for the OS
    but most likely has nothing to do with your issue. Raid 1 gives you
    redundancy and Raid 0 doubles your chance of failure!!!
    http://www.m-techlaptops.com/raid.htm
     
    Frank McCallister SBS MVP, Jan 18, 2008
    #9
  10. Tom Schwartz

    Tom Schwartz Guest

    Frank, Thanks for the ideas about the NIC and Post SP2 updates but I have
    checked and all seems to be in order there. Raid 0 was a complete bumble on
    my part should have read Raid 1 so thanks again for catching that.
    Currently Running backup to Drive D so will see if anything breaks.....
     
    Tom Schwartz, Jan 19, 2008
    #10
  11. Tom Schwartz

    Tom Schwartz Guest

    Okay Gents,

    After I have moved the Backup location around to a multitude of locations
    including:
    1. Drive C (Raid 1)
    2. Drive D (Raid 5)
    3. USB Drive
    4. Network Share

    It appears that the destination of the backup is not the problem.
    When the NTBackup fails at a random day, the problem normally occurres at
    somepoint in backing up the Information Stores of Exchange. I have
    experimented with ISINTEG and ESEUTIL /D in order to find out if I have any
    corruption or if defragging the database has any impact on performance or
    lockup of the SERVER.

    What I have found is that if I ISINTEG -FIX either the priv1.edb or pub1.edb
    files, mount the store, dismount the store, I always have a couple errors to
    fix in the files. IS THIS NORMAL or do I have some deeper issue that ISINTEG
    says it fixes but after a mount / dismount sequence the problem shows up
    again.

    I really am starting to think my "UNRESPONSIVE SYSTEM" is because NTBACKUP
    is hanging due to a number of error in the Information Store files but I
    can't prove it.

    Any Fixes or Troubleshooting Ideas?

    Tom
     
    Tom Schwartz, Feb 9, 2008
    #11
  12. It's highly unusual for the exchange database to become corrupted, except in
    instances where A/V file scanning is happening, and/or there are hardware
    issues with disks or ram.
     
    Les Connor [SBS MVP], Feb 9, 2008
    #12
  13. Tom Schwartz

    John Pare Guest

    I have had this exact problem with a clients Dell PE2900 server. The only new information that I have is that I can cause the problem to not occur if I move the start of the backup from 11:00 P.M. to 9:00 P.M.
    This is the first thing that I have tried and it seems to have cleared it up. I am suspect of the failure time which occurred during the beginning of the information store maintenance period, around 12:30 A.M., and coincident with the start of the NTbackup verification.
    This is the first thing that I have tried in order to keep the customer going. Otherwise the server hangs and has to be forcibly restarted.
     
    John Pare, Nov 24, 2008
    #13
  14. Tom Schwartz

    John Pare Guest

     
    John Pare, Dec 5, 2008
    #14
  15. Tom Schwartz

    John Pare Guest

    Sory, I was replying to the following post:

    Question
    Ever since upgrading to a Dell 2900 PowerEdge server, last year, I've had
    problems with this box hanging during backups.

    It isn't the backup itself that causes hanging, but the server stops
    responding during verfication.

    There are a number points about this, which I beleive are relevant;

    1. There are no events logged in the event log.

    2. The 'Begin Verification' event logged but not the 'End'.

    3. The Backup log for the crashed job is truncated, and the Verification
    section is missing.

    4. The incidence of this issue hasincreased dramitically since I installed
    Symantec EndPoint Protection Manager (11.0) on this box. So it must be
    related to AV software.

    5. I'm backing up to an external USB drive.

    Anyone got any ideas about how to fix this issue?

    Thanks in anticipation,
     
    John Pare, Dec 5, 2008
    #15
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.