How soon before you deploy patches in a commercial environment

I note that there was previously a thread (some time in 2006 "what are
general steps to test new released updates") where a few people
responded suggesting that a less than a day's testing be applied
before deployment - but generally critical updates should be deployed
immediately.

I'm interested to know if any other corporates (with, let's say 1000+
systems) deploy updates within a day to all production systems.

I was working with a Mid-Sized corporate (approx 1000 system) who
still choose to deploy updates a month after they are released - this
time is used for soak testing the updates with a test user group.

Am I wrong to be concerned about this - it seems overly long?

At the same time they can point to the fact that they do not appear to
have suffered an exploit - which might partly be luck or might be due
to their AV and Web blocking systems stopping the ingress of known
malware. Or the design of their infrastructure (proxy etc) which will
stop some of these canned exploits from working anyway (anything
wanting to talk to the mother-ship on an unusual port will fail).

They choose to test for this a monthto allow time for other companies/
individuals to find the problesm in updates and alert the community
and/or for MS to recall flawed updates (which seems to happen quite a
bit). The argument being that a flawed update could cause just as much
disruption as an exploit and a one hour test might not be sufficient
to identify such a flaw.

Now, most of my work is with smaller companies who update almost
immediately, but then again, if the update is flawed they can also
recover the situration much more quickly - so should I be pressing
home my argument with the larger organisation or is this a fairly
standard approach.

The trouble is - it doesn't really matter whether it's a week or a
month (both are too long) the only real value is if they patch within
a day, but that does leave them wide open to dodgy updates...

Answers, more like opinions, inline:

wrote:

> I note that there was previously a thread (some time in 2006 "what are
> general steps to test new released updates") where a few people
> responded suggesting that a less than a day's testing be applied
> before deployment - but generally critical updates should be deployed
> immediately.
>
> I'm interested to know if any other corporates (with, let's say 1000+
> systems) deploy updates within a day to all production systems.

I doubt that most large-sized corps deploy updates within a day. As you
state later, the risk from deploying a 'toxic' update would equal or
exceed the damage from malware.
Of course, what *kind* of business is a critical factor. If it's a
business that can not allow a breach of personal data, then deploying
updates as fast as possible is *critical*.


> I was working with a Mid-Sized corporate (approx 1000 system) who
> still choose to deploy updates a month after they are released - this
> time is used for soak testing the updates with a test user group.
>
> Am I wrong to be concerned about this - it seems overly long?

Yes, your concern is valid. Given today's state of the net and the way
bad actors reverse-engineer security updates, a month is too long for
critical security updates.

>
> At the same time they can point to the fact that they do not appear to
> have suffered an exploit - which might partly be luck or might be due
> to their AV and Web blocking systems stopping the ingress of known
> malware. Or the design of their infrastructure (proxy etc) which will
> stop some of these canned exploits from working anyway (anything
> wanting to talk to the mother-ship on an unusual port will fail).
>
> They choose to test for this a monthto allow time for other companies/
> individuals to find the problesm in updates and alert the community
> and/or for MS to recall flawed updates (which seems to happen quite a
> bit). The argument being that a flawed update could cause just as much
> disruption as an exploit and a one hour test might not be sufficient
> to identify such a flaw.

It appears that the company has a solid security practice. BUT, it only
takes one well crafted exploit to defeat said sec practice. Again, this
would be more critical for a business that retains personal data.
When there are issues with updates it does not take more than one week
for said issues to emerge. In all the time that I've participated in
this newsgroup, and that's 8 years, I can honestly state that out of all
the updates MS has issued there have only been around 3% of the updates
that have caused issues. MS *rarely* has to release an updated update
due to issues. In fact, most of the issues are directly related to ....
* 3rd party software *.

>
> Now, most of my work is with smaller companies who update almost
> immediately, but then again, if the update is flawed they can also
> recover the situration much more quickly - so should I be pressing
> home my argument with the larger organisation or is this a fairly
> standard approach.
>
> The trouble is - it doesn't really matter whether it's a week or a
> month (both are too long) the only real value is if they patch within
> a day, but that does leave them wide open to dodgy updates...
>


Generally speaking, smaller organizations do not have the resources to
implement sound security practices. And, the installation of a 'toxic'
update is much quicker to recover from. Of course that's based on a back
up and image strategy. But even a flatten and reload is still not as
painful as it is for larger organizations.
IMHO, a month is way too long to wait to install a critical security
update. One day the organization will lose the race with the
reverse-engineering bad guyz.
So, you have to demonstrate to management the cost of testing and
deploying critical sec updates vs. the cost of being exploited by not
doing so expeditiously.

Another issue, which is fortunately disapperaing, is that smaller
companies who do deploy the updates faster do not report issues to MS.
The quicker MS hears about said issues the faster they are to react and
diagnose just what is causing the issues. Thankfully, more and more
small companies and individuals are reporting the issues expeditiously
to MS. And that's good for everyone <w>


MowGreen [MVP 2003-2008]
===============
*-343-* FDNY
Never Forgotten
===============