SVCHost wuauserv 100% cpu consumption

serious issue in production environment with line systems suddently hung from
windows update.

Scenario:
A few weeks ago coworkers updated all WSUS servers to 3.0
First patch cycle since the WSUS server upgrade.
Latest updates approved and production systems (auto download manual
install) downloaded new patches on Friday.
Monday evening some of the servers/clients started hanging up due to
wuauserv/svchost process consuming the cpu without any action being taken.
Several hours later and the following day other systems started experiencing
the same issues.
The GPOs for the WSUS settings were disabled and the systems restarted to
resolve the immediate issue.
Systems w/problem all appear to have version 7.0.6000.374 of the
wuaueng.dll. Some systems with this version did not display the problem.
Office is NOT installed on some of the systems w/problem.
All systems do have KB927891 Installed
In testing applying lates WindowsUpdateAgent30 which brings wuaueng.dll to
version 7.2.6001.788 has resolved the issue. Although we are not sure we are
attacking the right versioning problem or resolution.
The only actions taken on Monday was by an admin that ran a "cleanup" on the
master WSUS server.

What happened on Monday to cause this to all of sudden start happening?
Is there debugging we can do to find the exact cause of the problem?

At this point the problem is still rampant and WU has become a problem with
our IT Teams and our Process Control people. Without being able to determine
the exact cause and a good resolution this impacts the future of WSUS in our
environment.

From the WindowsUpdate.log of one of the machines affected


2009-07-27 00:32:10:551 1068 650 AU #############
2009-07-27 00:32:10:551 1068 650 AU ## START ## AU: Search for updates
2009-07-27 00:32:10:551 1068 650 AU #########
2009-07-27 00:32:10:551 1068 650 AU <<## SUBMITTED ## AU: Search for updates
[CallId = {23837696-05A0-4547-B357-47D768E3D3B2}]
2009-07-27 00:32:10:551 1068 660 Agent *************
2009-07-27 00:32:10:551 1068 660 Agent ** START ** Agent: Finding updates
[CallerId = AutomaticUpdates]
2009-07-27 00:32:10:551 1068 660 Agent *********
2009-07-27 00:32:10:551 1068 660 Agent * Online = Yes; Ignore download
priority = No
2009-07-27 00:32:10:551 1068 660 Agent * Criteria = "IsHidden=0 and
IsInstalled=0 and DeploymentAction='Installation' and IsAssigned=1 or
IsHidden=0 and IsPresent=1 and DeploymentAction='Uninstallation' and
IsAssigned=1 or IsHidden=0 and IsInstalled=1 and
DeploymentAction='Installation' and IsAssigned=1 and RebootRequired=1 or
IsHidden=0 and IsInstalled=0 and DeploymentAction='Uninstallation' and
IsAssigned=1 and RebootRequired=1"
2009-07-27 00:32:10:551 1068 660 Agent * ServiceID =
{3DA21691-E39D-4DA6-8A4B-B43877BCB1B7}
2009-07-27 00:32:10:551 1068 660 Misc Validating signature for
C:\WINDOWS\SoftwareDistribution\SelfUpdate\Default \wuident.cab:
2009-07-27 00:32:10:551 1068 660 Misc Microsoft signed: Yes
2009-07-27 00:32:12:817 1068 660 Misc Validating signature for
C:\WINDOWS\SoftwareDistribution\SelfUpdate\Default \wuident.cab:
2009-07-27 00:32:12:817 1068 660 Misc Microsoft signed: Yes
2009-07-27 00:32:17:332 1068 660 PT +++++++++++ PT: Synchronizing server
updates +++++++++++
2009-07-27 00:32:17:332 1068 660 PT + ServiceId =
{3DA21691-E39D-4DA6-8A4B-B43877BCB1B7}, Server URL =
http://garyworks-sus.gry.uss.com/Cli...ce/client.asmx
2009-07-27 00:32:17:488 1068 660 PT WARNING: Cached cookie has expired or
new PID is available
2009-07-27 00:32:17:488 1068 660 PT Initializing simple targeting cookie,
clientId = 514b0c27-1a03-4dc4-b44d-3a2013f26664, target group = Production,
DNS name = gwpc-anc2-ent.gry.uss.com
2009-07-27 00:32:17:488 1068 660 PT Server URL =
http://garyworks-sus.gry.uss.com/Sim...impleAuth.asmx
2009-07-27 00:32:17:504 1068 660 PT WARNING: GetCookie failure, error =
0x8024400D, soap client error = 7, soap error code = 300, HTTP status code =
200
2009-07-27 00:32:17:504 1068 660 PT WARNING: SOAP Fault: 0x00012c
2009-07-27 00:32:17:504 1068 660 PT WARNING: faultstring:Fault occurred
2009-07-27 00:32:17:504 1068 660 PT WARNING: ErrorCode:ConfigChanged(2)
2009-07-27 00:32:17:504 1068 660 PT WARNING: Messagenull)
2009-07-27 00:32:17:504 1068 660 PT WARNING:
Method:"http://www.microsoft.com/SoftwareDistribution/Server/ClientWebService/GetCookie"
2009-07-27 00:32:17:504 1068 660 PT WARNING:
ID:f9a32825-870e-48ae-a665-7d8449afdd00
2009-07-27 00:32:17:551 1068 660 PT WARNING: Cached cookie has expired or
new PID is available
2009-07-27 00:32:17:551 1068 660 PT Initializing simple targeting cookie,
clientId = 514b0c27-1a03-4dc4-b44d-3a2013f26664, target group = Production,
DNS name = gwpc-anc2-ent.gry.uss.com
2009-07-27 00:32:17:551 1068 660 PT Server URL =
http://garyworks-sus.gry.uss.com/Sim...impleAuth.asmx
2009-07-27 00:32:21:785 1068 660 PT +++++++++++ PT: Synchronizing extended
update info +++++++++++
2009-07-27 00:32:21:785 1068 660 PT + ServiceId =
{3DA21691-E39D-4DA6-8A4B-B43877BCB1B7}, Server URL =
http://garyworks-sus.gry.uss.com/Cli...ce/client.asmx
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{8079EF5C-B355-4C65-8888-8D708BEB2D87}.103 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{90030620-A692-4641-9166-B7C4AEE44B86}.101 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{7FE46519-51F4-4F67-A74F-2DE851A1DDB5}.101 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{69794AD5-4C95-4904-8BDA-B308BB9663D4}.101 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{8A156488-B91D-4D21-A455-50203968ADCB}.100 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{0975455F-D040-4EEC-9013-81D19BF3DD0A}.102 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{F1E485A6-8B18-4D1A-B1B2-08856E3A9B70}.102 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{1406378F-BA2C-40FA-816C-922C846FF5E5}.101 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{00D97853-4261-4FC6-97E6-28A9383CC8B5}.101 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{310A4C08-593D-41A3-BB57-83B386D773B5}.103 to search result
2009-07-27 00:32:28:801 1068 660 Agent Update
{1D78D4F2-F09A-49DC-AC5B-F3079E83AD18}.101 is pruned out due to potential
supersedence
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{DF2F0A98-6E35-4379-AB33-A0303EF74B2A}.102 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{0F4FB382-5878-48C5-B7A4-C70F8EB0D7C6}.102 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{C4640A2A-B0E1-4F85-BEFA-300AB95699EB}.101 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{0F9C03F7-D1E1-4D8F-9B66-8B1C7D30421F}.101 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{1BAC6A90-E250-4F1E-B254-5D076EFB2966}.100 to search result
2009-07-27 00:32:28:801 1068 660 Agent Update
{BEEAF67F-53D9-46E2-8C06-DDF108D5D631}.100 is pruned out due to potential
supersedence
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{0D841D92-DCFD-42D0-99CC-DA46ABBFB86C}.101 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{969D5F26-8B9A-418D-A8C4-DC6912FAEC43}.101 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{723B4632-B686-45B3-8AA5-6E638E598657}.101 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{15370190-2934-407F-BB3A-795CF52EF6F3}.101 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{42564933-0432-42EC-A399-3EBDD4D73ED0}.100 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{4E41018C-72E6-4F63-9F05-AC088F1352F8}.101 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{19986D1F-0652-4816-8D54-0ADD7D1D444C}.102 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Added update
{6FF1BD9D-C442-4C06-93F3-A768877A5254}.101 to search result
2009-07-27 00:32:28:801 1068 660 Agent * Found 23 updates and 52
categories in search; evaluated appl. rules of 933 out of 1372 deployed
entities
2009-07-27 00:32:28:863 1068 660 Agent *********
2009-07-27 00:32:28:863 1068 660 Agent ** END ** Agent: Finding updates
[CallerId = AutomaticUpdates]
2009-07-27 00:32:28:863 1068 660 Agent *************
2009-07-27 00:32:28:863 1068 660 Report REPORT EVENT:
{F8F92CFD-E71E-4D45-B2DC-A24AB556680F} 2009-07-27
00:32:28:863-0500 1 147 101 {00000000-0000-0000-0000-000000000000} 0 0 AutomaticUpdates Success Software
Synchronization Windows Update Client successfully detected 23 updates.
2009-07-27 00:32:28:863 1068 16c AU >>## RESUMED ## AU: Search for updates
[CallId = {23837696-05A0-4547-B357-47D768E3D3B2}]
2009-07-27 00:32:28:863 1068 16c AU # 23 updates detected
2009-07-27 00:32:28:863 1068 660 Report REPORT EVENT:
{529A6A03-32C8-4105-8044-B689B5D93EC3} 2009-07-27
00:32:28:863-0500 1 156 101 {00000000-0000-0000-0000-000000000000} 0 0 AutomaticUpdates Success Pre-Deployment Check Reporting client status.
2009-07-27 00:32:28:879 1068 16c AU #########
2009-07-27 00:32:28:879 1068 16c AU ## END ## AU: Search for updates
[CallId = {23837696-05A0-4547-B357-47D768E3D3B2}]
2009-07-27 00:32:28:879 1068 16c AU #############
2009-07-27 00:32:28:879 1068 16c AU AU setting next detection timeout to
2009-07-27 16:57:01
2009-07-27 00:46:43:655 1068 660 Report Uploading 2 events using cached
cookie, reporting URL =
http://garyworks-sus.gry.uss.com/Rep...ebService.asmx
2009-07-27 00:46:43:655 1068 660 Report Reporter successfully uploaded 2
events.
2009-07-27 05:21:44:235 1068 650 AU AU received policy change subscription
event
2009-07-27 11:58:04:023 1068 650 AU #############
2009-07-27 11:58:04:023 1068 650 AU ## START ## AU: Search for updates
2009-07-27 11:58:04:023 1068 650 AU #########
2009-07-27 11:58:04:023 1068 650 AU <<## SUBMITTED ## AU: Search for updates
[CallId = {3963CB62-CE50-453B-9A22-8037459B3B44}]
2009-07-27 11:58:04:023 1068 ec Agent *************
2009-07-27 11:58:04:023 1068 ec Agent ** START ** Agent: Finding updates
[CallerId = AutomaticUpdates]
2009-07-27 11:58:04:023 1068 ec Agent *********
2009-07-27 11:58:04:023 1068 ec Agent * Online = Yes; Ignore download
priority = No
2009-07-27 11:58:04:023 1068 ec Agent * Criteria = "IsHidden=0 and
IsInstalled=0 and DeploymentAction='Installation' and IsAssigned=1 or
IsHidden=0 and IsPresent=1 and DeploymentAction='Uninstallation' and
IsAssigned=1 or IsHidden=0 and IsInstalled=1 and
DeploymentAction='Installation' and IsAssigned=1 and RebootRequired=1 or
IsHidden=0 and IsInstalled=0 and DeploymentAction='Uninstallation' and
IsAssigned=1 and RebootRequired=1"
2009-07-27 11:58:04:023 1068 ec Agent * ServiceID =
{3DA21691-E39D-4DA6-8A4B-B43877BCB1B7}
2009-07-27 11:58:04:023 1068 ec Misc Validating signature for
C:\WINDOWS\SoftwareDistribution\SelfUpdate\Default \wuident.cab:
2009-07-27 11:58:04:023 1068 ec Misc Microsoft signed: Yes
2009-07-27 11:58:06:295 1068 ec Misc Validating signature for
C:\WINDOWS\SoftwareDistribution\SelfUpdate\Default \wuident.cab:
2009-07-27 11:58:06:295 1068 ec Misc Microsoft signed: Yes
2009-07-27 11:58:10:884 1068 ec PT +++++++++++ PT: Synchronizing server
updates +++++++++++
2009-07-27 11:58:10:884 1068 ec PT + ServiceId =
{3DA21691-E39D-4DA6-8A4B-B43877BCB1B7}, Server URL =
http://garyworks-sus.gry.uss.com/Cli...ce/client.asmx
2009-07-27 11:58:11:041 1068 ec PT WARNING: Cached cookie has expired or
new PID is available
2009-07-27 11:58:11:041 1068 ec PT Initializing simple targeting cookie,
clientId = 514b0c27-1a03-4dc4-b44d-3a2013f26664, target group = Production,
DNS name = gwpc-anc2-ent.gry.uss.com
2009-07-27 11:58:11:041 1068 ec PT Server URL =
http://garyworks-sus.gry.uss.com/Sim...impleAuth.asmx
2009-07-27 11:58:15:333 1068 ec PT +++++++++++ PT: Synchronizing extended
update info +++++++++++
2009-07-27 11:58:15:333 1068 ec PT + ServiceId =
{3DA21691-E39D-4DA6-8A4B-B43877BCB1B7}, Server URL =
http://garyworks-sus.gry.uss.com/Cli...ce/client.asmx
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{8079EF5C-B355-4C65-8888-8D708BEB2D87}.103 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{90030620-A692-4641-9166-B7C4AEE44B86}.101 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{7FE46519-51F4-4F67-A74F-2DE851A1DDB5}.101 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{69794AD5-4C95-4904-8BDA-B308BB9663D4}.101 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{8A156488-B91D-4D21-A455-50203968ADCB}.100 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{0975455F-D040-4EEC-9013-81D19BF3DD0A}.102 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{F1E485A6-8B18-4D1A-B1B2-08856E3A9B70}.102 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{1406378F-BA2C-40FA-816C-922C846FF5E5}.101 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{00D97853-4261-4FC6-97E6-28A9383CC8B5}.101 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{310A4C08-593D-41A3-BB57-83B386D773B5}.103 to search result
2009-07-27 11:58:22:460 1068 ec Agent Update
{1D78D4F2-F09A-49DC-AC5B-F3079E83AD18}.101 is pruned out due to potential
supersedence
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{DF2F0A98-6E35-4379-AB33-A0303EF74B2A}.102 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{0F4FB382-5878-48C5-B7A4-C70F8EB0D7C6}.102 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{C4640A2A-B0E1-4F85-BEFA-300AB95699EB}.101 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{0F9C03F7-D1E1-4D8F-9B66-8B1C7D30421F}.101 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{1BAC6A90-E250-4F1E-B254-5D076EFB2966}.100 to search result
2009-07-27 11:58:22:460 1068 ec Agent Update
{BEEAF67F-53D9-46E2-8C06-DDF108D5D631}.100 is pruned out due to potential
supersedence
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{0D841D92-DCFD-42D0-99CC-DA46ABBFB86C}.101 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{969D5F26-8B9A-418D-A8C4-DC6912FAEC43}.101 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{723B4632-B686-45B3-8AA5-6E638E598657}.101 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{15370190-2934-407F-BB3A-795CF52EF6F3}.101 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{42564933-0432-42EC-A399-3EBDD4D73ED0}.100 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{4E41018C-72E6-4F63-9F05-AC088F1352F8}.101 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{19986D1F-0652-4816-8D54-0ADD7D1D444C}.102 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Added update
{6FF1BD9D-C442-4C06-93F3-A768877A5254}.101 to search result
2009-07-27 11:58:22:460 1068 ec Agent * Found 23 updates and 52
categories in search; evaluated appl. rules of 933 out of 1372 deployed
entities
2009-07-27 11:58:22:554 1068 ec Agent *********
2009-07-27 11:58:22:554 1068 ec Agent ** END ** Agent: Finding updates
[CallerId = AutomaticUpdates]
2009-07-27 11:58:22:554 1068 ec Agent *************
2009-07-27 11:58:22:554 1068 ec Report REPORT EVENT:
{3762E18A-4A72-4C07-A7B4-7FE97C2BF127} 2009-07-27
11:58:22:522-0500 1 147 101 {00000000-0000-0000-0000-000000000000} 0 0 AutomaticUpdates Success Software
Synchronization Windows Update Client successfully detected 23 updates.
2009-07-27 11:58:22:554 1068 d68 AU >>## RESUMED ## AU: Search for updates
[CallId = {3963CB62-CE50-453B-9A22-8037459B3B44}]
2009-07-27 11:58:22:554 1068 d68 AU # 23 updates detected
2009-07-27 11:58:22:569 1068 ec Report REPORT EVENT:
{4A32ECB9-8B2A-44DB-B237-38DF1D603BEA} 2009-07-27
11:58:22:554-0500 1 156 101 {00000000-0000-0000-0000-000000000000} 0 0 AutomaticUpdates Success Pre-Deployment Check Reporting client status.
2009-07-27 11:58:22:569 1068 d68 AU #########
2009-07-27 11:58:22:569 1068 d68 AU ## END ## AU: Search for updates
[CallId = {3963CB62-CE50-453B-9A22-8037459B3B44}]
2009-07-27 11:58:22:569 1068 d68 AU #############
2009-07-27 11:58:22:569 1068 d68 AU AU setting next detection timeout to
2009-07-28 03:05:36
2009-07-27 12:12:39:500 1068 ec Report Uploading 2 events using cached
cookie, reporting URL =
http://garyworks-sus.gry.uss.com/Rep...ebService.asmx
2009-07-27 12:12:39:516 1068 ec Report Reporter successfully uploaded 2
events.
2009-07-27 21:26:17:491 1068 650 AU AU received policy change subscription
event
2009-07-27 22:05:36:628 1068 650 AU #############
2009-07-27 22:05:36:628 1068 650 AU ## START ## AU: Search for updates
2009-07-27 22:05:36:628 1068 650 AU #########
2009-07-27 22:05:36:628 1068 650 AU <<## SUBMITTED ## AU: Search for updates
[CallId = {9E00ACE0-4EE1-4CA5-B030-6920DC0AFFA6}]
2009-07-27 22:05:36:628 1068 a18 Agent *************
2009-07-27 22:05:36:628 1068 a18 Agent ** START ** Agent: Finding updates
[CallerId = AutomaticUpdates]
2009-07-27 22:05:36:628 1068 a18 Agent *********
2009-07-27 22:05:36:628 1068 a18 Agent * Online = Yes; Ignore download
priority = No
2009-07-27 22:05:36:628 1068 a18 Agent * Criteria = "IsHidden=0 and
IsInstalled=0 and DeploymentAction='Installation' and IsAssigned=1 or
IsHidden=0 and IsPresent=1 and DeploymentAction='Uninstallation' and
IsAssigned=1 or IsHidden=0 and IsInstalled=1 and
DeploymentAction='Installation' and IsAssigned=1 and RebootRequired=1 or
IsHidden=0 and IsInstalled=0 and DeploymentAction='Uninstallation' and
IsAssigned=1 and RebootRequired=1"
2009-07-27 22:05:36:628 1068 a18 Agent * ServiceID =
{3DA21691-E39D-4DA6-8A4B-B43877BCB1B7}
2009-07-27 22:05:36:628 1068 a18 Misc Validating signature for
C:\WINDOWS\SoftwareDistribution\SelfUpdate\Default \wuident.cab:
2009-07-27 22:05:36:628 1068 a18 Misc Microsoft signed: Yes
2009-07-27 22:05:38:893 1068 a18 Misc Validating signature for
C:\WINDOWS\SoftwareDistribution\SelfUpdate\Default \wuident.cab:
2009-07-27 22:05:38:893 1068 a18 Misc Microsoft signed: Yes
2009-07-27 22:05:43:878 1068 a18 PT +++++++++++ PT: Synchronizing server
updates +++++++++++
2009-07-27 22:05:43:878 1068 a18 PT + ServiceId =
{3DA21691-E39D-4DA6-8A4B-B43877BCB1B7}, Server URL =
http://garyworks-sus.gry.uss.com/Cli...ce/client.asmx
2009-07-27 22:05:44:050 1068 a18 PT WARNING: Cached cookie has expired or
new PID is available
2009-07-27 22:05:44:050 1068 a18 PT Initializing simple targeting cookie,
clientId = 514b0c27-1a03-4dc4-b44d-3a2013f26664, target group = Production,
DNS name = gwpc-anc2-ent.gry.uss.com
2009-07-27 22:05:44:050 1068 a18 PT Server URL =
http://garyworks-sus.gry.uss.com/Sim...impleAuth.asmx
2009-07-27 22:05:44:065 1068 a18 PT WARNING: GetCookie failure, error =
0x8024400D, soap client error = 7, soap error code = 300, HTTP status code =
200
2009-07-27 22:05:44:065 1068 a18 PT WARNING: SOAP Fault: 0x00012c
2009-07-27 22:05:44:065 1068 a18 PT WARNING: faultstring:Fault occurred
2009-07-27 22:05:44:065 1068 a18 PT WARNING: ErrorCode:ConfigChanged(2)
2009-07-27 22:05:44:065 1068 a18 PT WARNING: Messagenull)
2009-07-27 22:05:44:065 1068 a18 PT WARNING:
Method:"http://www.microsoft.com/SoftwareDistribution/Server/ClientWebService/GetCookie"
2009-07-27 22:05:44:065 1068 a18 PT WARNING:
ID:53cd94a0-6dee-4795-a703-a1b757d32025
2009-07-27 22:05:44:097 1068 a18 PT WARNING: Cached cookie has expired or
new PID is available
2009-07-27 22:05:44:097 1068 a18 PT Initializing simple targeting cookie,
clientId = 514b0c27-1a03-4dc4-b44d-3a2013f26664, target group = Production,
DNS name = gwpc-anc2-ent.gry.uss.com
2009-07-27 22:05:44:097 1068 a18 PT Server URL =
http://garyworks-sus.gry.uss.com/Sim...impleAuth.asmx
2009-07-27 22:05:57:222 1068 a18 PT +++++++++++ PT: Synchronizing extended
update info +++++++++++
2009-07-27 22:05:57:222 1068 a18 PT + ServiceId =
{3DA21691-E39D-4DA6-8A4B-B43877BCB1B7}, Server URL =
http://garyworks-sus.gry.uss.com/Cli...ce/client.asmx
2009-07-28 00:29:30:140 1072 3a0 Misc =========== Logging initialized
(build: 7.0.6000.374, tz: -0500) ===========
2009-07-28 00:29:30:140 1072 3a0 Misc = Process:
C:\WINDOWS\System32\svchost.exe
2009-07-28 00:29:30:140 1072 3a0 Misc = Module:
C:\WINDOWS\system32\wuaueng.dll
2009-07-28 00:29:30:140 1072 3a0 Service *************
2009-07-28 00:29:30:140 1072 3a0 Service ** START ** Service: Service startup
2009-07-28 00:29:30:140 1072 3a0 Service *********
2009-07-28 00:29:30:234 1072 3a0 Agent * WU client version 7.0.6000.374
2009-07-28 00:29:30:234 1072 3a0 Agent * Base directory:
C:\WINDOWS\SoftwareDistribution
2009-07-28 00:29:30:250 1072 3a0 Agent * Access type: No proxy
2009-07-28 00:29:30:250 1072 3a0 Agent * Network state: Connected
2009-07-28 00:30:18:016 1072 3a0 Agent *********** Agent: Initializing
Windows Update Agent ***********
2009-07-28 00:30:18:016 1072 3a0 Agent *********** Agent: Initializing
global settings cache ***********
2009-07-28 00:30:18:016 1072 3a0 Agent * WSUS server:
http://garyworks-sus.gry.uss.com
2009-07-28 00:30:18:016 1072 3a0 Agent * WSUS status server:
http://garyworks-sus.gry.uss.com
2009-07-28 00:30:18:016 1072 3a0 Agent * Target group: Production
2009-07-28 00:30:18:016 1072 3a0 Agent * Windows Update access disabled: No
2009-07-28 00:30:21:533 1072 3a0 DnldMgr Download manager restoring 0
downloads
2009-07-28 00:30:21:568 1072 3a0 AU ########### AU: Initializing Automatic
Updates ###########
2009-07-28 00:30:21:568 1072 3a0 AU # AU disabled through Policy
2009-07-28 00:30:21:602 1072 3a0 Agent Switching to hardware-verified
ClientId.
2009-07-28 00:30:21:602 1072 3a0 AU AU finished delayed initialization
2009-07-28 00:30:22:592 1072 3a0 Agent Created new random SusClientId
2c975b02-e8fe-4d91-a912-bce75d99495a. Old Id:
86d200eb-9312-4828-b412-48f8075b6753.
2009-07-28 00:30:22:592 1072 3a0 Report *********** Report: Initializing
static reporting data ***********
2009-07-28 00:30:22:592 1072 3a0 Report * OS Version = 5.1.2600.2.0.65792
2009-07-28 00:30:22:661 1072 3a0 Report * Computer Brand = Hewlett-Packard
2009-07-28 00:30:22:661 1072 3a0 Report * Computer Model = HP dc5000
uT(PB466A)
2009-07-28 00:30:22:678 1072 3a0 Report * Bios Revision = 786B0 v1.00
2009-07-28 00:30:22:678 1072 3a0 Report * Bios Name = 786B0 v1.00
2009-07-28 00:30:22:678 1072 3a0 Report * Bios Release Date =
2004-02-12T00:00:00
2009-07-28 00:30:22:678 1072 3a0 Report * Locale ID = 1033
2009-07-28 18:12:26:202 1072 3a0 AU AU received policy change subscription
event

"Jen" <> wrote in message
news:FD3FA574-090A-4C91-ACE5-...

> Systems w/problem all appear to have version 7.0.6000.374 of the
> wuaueng.dll.

This would suggest that you installed the WSUS v3.0 =RTM= package, not WSUS
v3.0 SP1, which would have updated the clients to the v7.1 WUAgent, which
will overcome some of these performance issues.

> In testing applying lates WindowsUpdateAgent30 which brings wuaueng.dll to
> version 7.2.6001.788 has resolved the issue.

Yes, this is a known resolution. The v7.2 WUAgent has specific modifications
designed to deal with update servers with large catalogs.

> The only actions taken on Monday was by an admin that ran a "cleanup" on
> the
> master WSUS server.

This is a good start. Combined with that "cleanup" verify that you've
declined all superseded (unneeded) updates. As noted, the performance issues
are typically caused by a large catalog, and removing the unneeded updates
from the list of updates the WUAgent needs to scan can help significantly.

> What happened on Monday to cause this to all of sudden start happening?

You upgraded to WSUS 3.0 (RTM, it appears), which also caused the database
to be migrated, which may have introduced all sorts of variables impacting
the performance of the server.

> Is there debugging we can do to find the exact cause of the problem?

The exact cause(s) are well known.

> At this point the problem is still rampant and WU has become a problem
> with
> our IT Teams and our Process Control people. Without being able to
> determine
> the exact cause and a good resolution this impacts the future of WSUS in
> our
> environment.

Basically three solutions:

1. Upgrade the WSUS v3 server to WSUS v3 Service Pack 1, which will deploy
the v7.1 WUAgent to all of your systems. This will help overcome known
issues with the v7.0 WUAgent.

2. Ensure you've performend necessary updates maintenance on your WSUS
servers -- specifically with respect to declining superseded updates.

3. Deploy the v7.2 WUAgent

In addiiton, WSUS v3 Service Pack 2 (Release Candidate) contains a newer
update to the WUAgent (v7.4), and will be released shortly. The v7.4 WUAgent
also addresses performance issues known to exist in the v7.1 WUAgent, and
I'd suggest being ready to deploy WSUS Service Pack 2 the moment it's
released.



--
Lawrence Garvin, M.S., MCITP:EA, MCDBA
Principal/CTO, Onsite Technology Solutions, Houston, Texas
Microsoft MVP - Software Distribution (2005-2009)

MS WSUS Website: http://www.microsoft.com/wsus
My MVP Profile: http://mvp.support.microsoft.com/pro...awrence.Garvin

"Jen" <> wrote in message
news:031D9CE1-80AC-4B33-AA32-...

> From the WindowsUpdate.log of one of the machines affected

This log is not showing any evidence of delayed scan times, which is one of
the known causes of the 100% utilization issue...

> 2009-07-27 00:32:17:488 1068 660 PT Server URL =
> http://garyworks-sus.gry.uss.com/Sim...impleAuth.asmx
> 2009-07-27 00:32:17:504 1068 660 PT WARNING: GetCookie failure, error =
> 0x8024400D, soap client error = 7, soap error code = 300, HTTP status code
> =
> 200
> 2009-07-27 00:32:17:504 1068 660 PT WARNING: SOAP Fault: 0x00012c
> 2009-07-27 00:32:17:504 1068 660 PT WARNING: faultstring:Fault
> occurred
> 2009-07-27 00:32:17:504 1068 660 PT WARNING:
> ErrorCode:ConfigChanged(2)
> 2009-07-27 00:32:17:504 1068 660 PT WARNING: Messagenull)

However, this scenario could be problematic. The first point of resolution,
though, would be to upgrade WSUS to SP1 and get the client upgraded to the
v7.1 WUAgent.


--
Lawrence Garvin, M.S., MCITP:EA, MCDBA
Principal/CTO, Onsite Technology Solutions, Houston, Texas
Microsoft MVP - Software Distribution (2005-2009)

MS WSUS Website: http://www.microsoft.com/wsus
My MVP Profile: http://mvp.support.microsoft.com/pro...awrence.Garvin

Because of the wide affect of this, the gpo is set to disable windows updates
for all of our production sites until 7.4 is available to be put on the WSUS
server so the clients can self update since the problem still appears with
7.1 and there is no good way to deploy the 7.2 version out to the production
systems running live processes.
(GPO would require a reboot which doesn't happen often enough on these
systems)

The hope is that we can put 7.4 on a WSUS server that doesnt have any
downloads/approvals, point all the production systems to it to get the new
agent and gradually move them back over to our production WSUS environment
(which we also plan to upgrade to 7.4)

We need to avoid any cpu hanging which can cause delays on the line and cost
the company money.

Question. Do you have an ETA for 7.4 (SP2)????


"Lawrence Garvin [MVP]" wrote:

> "Jen" <> wrote in message
> news:FD3FA574-090A-4C91-ACE5-...
>
> > Systems w/problem all appear to have version 7.0.6000.374 of the
> > wuaueng.dll.
>
> This would suggest that you installed the WSUS v3.0 =RTM= package, not WSUS
> v3.0 SP1, which would have updated the clients to the v7.1 WUAgent, which
> will overcome some of these performance issues.
>
> > In testing applying lates WindowsUpdateAgent30 which brings wuaueng.dll to
> > version 7.2.6001.788 has resolved the issue.
>
> Yes, this is a known resolution. The v7.2 WUAgent has specific modifications
> designed to deal with update servers with large catalogs.
>
> > The only actions taken on Monday was by an admin that ran a "cleanup" on
> > the
> > master WSUS server.
>
> This is a good start. Combined with that "cleanup" verify that you've
> declined all superseded (unneeded) updates. As noted, the performance issues
> are typically caused by a large catalog, and removing the unneeded updates
> from the list of updates the WUAgent needs to scan can help significantly.
>
> > What happened on Monday to cause this to all of sudden start happening?
>
> You upgraded to WSUS 3.0 (RTM, it appears), which also caused the database
> to be migrated, which may have introduced all sorts of variables impacting
> the performance of the server.
>
> > Is there debugging we can do to find the exact cause of the problem?
>
> The exact cause(s) are well known.
>
> > At this point the problem is still rampant and WU has become a problem
> > with
> > our IT Teams and our Process Control people. Without being able to
> > determine
> > the exact cause and a good resolution this impacts the future of WSUS in
> > our
> > environment.
>
> Basically three solutions:
>
> 1. Upgrade the WSUS v3 server to WSUS v3 Service Pack 1, which will deploy
> the v7.1 WUAgent to all of your systems. This will help overcome known
> issues with the v7.0 WUAgent.
>
> 2. Ensure you've performend necessary updates maintenance on your WSUS
> servers -- specifically with respect to declining superseded updates.
>
> 3. Deploy the v7.2 WUAgent
>
> In addiiton, WSUS v3 Service Pack 2 (Release Candidate) contains a newer
> update to the WUAgent (v7.4), and will be released shortly. The v7.4 WUAgent
> also addresses performance issues known to exist in the v7.1 WUAgent, and
> I'd suggest being ready to deploy WSUS Service Pack 2 the moment it's
> released.
>
>
>
> --
> Lawrence Garvin, M.S., MCITP:EA, MCDBA
> Principal/CTO, Onsite Technology Solutions, Houston, Texas
> Microsoft MVP - Software Distribution (2005-2009)
>
> MS WSUS Website: http://www.microsoft.com/wsus
> My MVP Profile: http://mvp.support.microsoft.com/pro...awrence.Garvin
>
>

"Jen" <> wrote in message
news:15596163-CA77-4AF9-8534-...
> Because of the wide affect of this, the gpo is set to disable windows
> updates
> for all of our production sites until 7.4 is available

That's a *BAD* decision, Jen. Particularly since an out-of-band security
update was just released two days ago. How do you plan to deploy that
update? Fixing the problem, as best you can now is a much better solution.

> there is no good way to deploy the 7.2 version out to the production
> systems running live processes.

Group Policy / Software Publishing is a good place to start.

> (GPO would require a reboot which doesn't happen often enough on these
> systems)

Installing the WUAgent does not require a system restart. Installing
WIndowsUpdateAgent30-x86.exe does not require a system restart.

> The hope is that we can put 7.4 on a WSUS server that doesnt have any
> downloads/approvals, point all the production systems to it to get the new
> agent and gradually move them back over to our production WSUS environment
> (which we also plan to upgrade to 7.4)

Then do that using the WSUS3SP2 RC today, if you wish. The WUAgent is not
likely to be updated between the RC and RTW, and even if it is, the
deployment will be triival.

> Question. Do you have an ETA for 7.4 (SP2)????

No.


--
Lawrence Garvin, M.S., MCITP:EA, MCDBA
Principal/CTO, Onsite Technology Solutions, Houston, Texas
Microsoft MVP - Software Distribution (2005-2009)

MS WSUS Website: http://www.microsoft.com/wsus
My MVP Profile: http://mvp.support.microsoft.com/pro...awrence.Garvin

the restart would be for the software install through a gpo to process.


"Lawrence Garvin [MVP]" wrote:

> "Jen" <> wrote in message
> news:15596163-CA77-4AF9-8534-...
> > Because of the wide affect of this, the gpo is set to disable windows
> > updates
> > for all of our production sites until 7.4 is available
>
> That's a *BAD* decision, Jen. Particularly since an out-of-band security
> update was just released two days ago. How do you plan to deploy that
> update? Fixing the problem, as best you can now is a much better solution.
>
> > there is no good way to deploy the 7.2 version out to the production
> > systems running live processes.
>
> Group Policy / Software Publishing is a good place to start.
>
> > (GPO would require a reboot which doesn't happen often enough on these
> > systems)
>
> Installing the WUAgent does not require a system restart. Installing
> WIndowsUpdateAgent30-x86.exe does not require a system restart.
>
> > The hope is that we can put 7.4 on a WSUS server that doesnt have any
> > downloads/approvals, point all the production systems to it to get the new
> > agent and gradually move them back over to our production WSUS environment
> > (which we also plan to upgrade to 7.4)
>
> Then do that using the WSUS3SP2 RC today, if you wish. The WUAgent is not
> likely to be updated between the RC and RTW, and even if it is, the
> deployment will be triival.
>
> > Question. Do you have an ETA for 7.4 (SP2)????
>
> No.
>
>
> --
> Lawrence Garvin, M.S., MCITP:EA, MCDBA
> Principal/CTO, Onsite Technology Solutions, Houston, Texas
> Microsoft MVP - Software Distribution (2005-2009)
>
> MS WSUS Website: http://www.microsoft.com/wsus
> My MVP Profile: http://mvp.support.microsoft.com/pro...awrence.Garvin
>
>

Jen wrote:

> the restart would be for the software install through a gpo to process.

psexec may be your friend here. Search for it on the Microsoft web site. It
allows you to run programs remotely.

Harry.