UPDATE – 12/4/2012: The System Center Configuration Manger team has released a hotfix (http://support.microsoft.com/kb/2783466) to address the re-downloading and redistributing issues described by this post. To minimize further impact, it is strongly suggested that all Configuration Manager 2007 admins install the hotfix immediately to all Configuration Manager 2007 primary and secondary site servers.
Hi all, Doug Neal and Erin Williams here. You may have seen the WSUS blog posted 31 October 2012 titled Support Tip: Many new revisions of updates may be downloaded by the WSUS server. We wanted to highlight that issue for you and also provide you with more information about the cause and what to expect going forward.
The first issue is related to digital certificates as described in KB2749655; the second is related to improvements in the Microsoft Update service used by Windows Server Update Services (WSUS), System Center Configuration Manager and Windows Intune as described in KB2718704.
ISSUE 1 - DIGITAL CERTIFICATES
The digital certificates issue is described in the MSRC advisory http://technet.microsoft.com/en-us/security/advisory/2749655 and the associated KB article http://support.microsoft.com/kb/2749655.
These updates released on October 9 (2nd Tuesday in October) and resulted in between 50 and 250 updates being changed depending on how many of these were on your servers. Some of these were revisions (metadata only changes). Some were re-releases (due to the code-signing elements being integrated into Windows CBS-based binaries). While the payload changed for some of these updates, none of them had functional or targeting changes beyond the signing corrections.
In the coming months, additional updates for this same issue will likely be released on future monthly 2nd Tuesdays and will appear as a similar set of 50 - 250 updates that are either revised, re-released or both. The impact on WSUS and Configuration Manager servers should be the same as they were on October 9. Windows Intune is not affected since it maintains the datastore in the cloud, not in a local database like WSUS and Configuration Manager servers.
ISSUE 2 - ADDITIONAL IMPROVEMENTS
As part of a strategy to improve the security of Windows/Microsoft Update, many updates were revised in other ways as mentioned in the MSRC blog http://blogs.technet.com/b/msrc/archive/2012/06/04/security-advisory-2718704-update-to-phased-mitigation-strategy.aspx, in the MSRC advisory http://technet.microsoft.com/en-us/security/advisory/2718704 and in the associated KB article http://support.microsoft.com/kb/2718704.
The WSUS team posted their related post Wednesday October 31: http://blogs.technet.com/b/sus/archive/2012/10/31/support-tip-many-new-revisions-of-updates-may-be-downloaded-by-the-wsus-server.aspx
Within the Microsoft Update (MU) service, a very large number of updates were improved in additional ways to secure and harden the service. The large number of improved updates became visible to WSUS servers on a rolling, one-time basis beginning the first week of October. This means that one WSUS admin may have received the improved revisions all at once one day after a sync, while another WSUS server may have received the same large batch of updates 1, 2, 5, 7 or even 14 days later than the earlier admin. Once these improvements come down to your WSUS or Configuration Manager server, you will not incur another experience like this again. This is a one-time sync of a large number of updates we've already made in our service - separate and different from those described in ISSUE 1 above.
For Configuration Manager admins, depending on how many of these improved updates were present in your WSUS server, you may have observed anywhere from 1000 or more revisions. As a result, your managed clients may have briefly indicated they weren't compliant (due to the new revisions). The latter issue will incur a one-time cost to re-download any active deployments to both sync and redistribute these to ConfigMgr distribution points.
Any deployment package that had an update in it that was revised will not deploy until the new update is manually downloaded. You may also experience deployment packages that have had the update contained within them removed. To resolve this issue simply re-download the affected update. Updates that are effected will have their deployment download status change from YES to NO as in the picture below. There may be bandwidth constraints during this re-downloading effort; the effort increases with the number of active deployments that need to be downloaded in your environment.
Click on above screenshot to enlarge
All WSUS servers will be impacted by these changes. WSUS servers that have KB2734608 applied will get the revisions immediately for patching Windows 8 and Windows Server 2012. MU is coordinating the rollout for other non KB2734608 patched servers.
In both cases, whether for digital certificates or the additional improvements, neither the targeting (metadata) nor payloads were changed in any functional way. While both changes improve the service for enterprises and consumers, the impact wasn't sufficiently understood beforehand and communicated proactively. We hope this explanation helps describe the situation and helps you plan for and accommodate these changes. We strive to provide a powerful service you can trust without interruption and we're already making improvements based on your feedback.
Doug Neal - Microsoft Update PM
Erin Williams – Configuration Manager Supportability PM
Yeah, on Saturday, our server said "Updated 11813 items in SMS database, new update source content version is 298" so...I'd say it's a little more than 50-250 items. With 360K clients, that's like 4.2Billion state messages that it generated. This issue has taken down our central for going on 4 days now with no end in sight. Unfortunately, what this does is make it so we'll never automatically sync again outside of patch tuesday since it has the ability to make CM completely unusable from the central. For big hierarchies, that's a big deal.
There, I've vented. I feel better.
What is the effect on clients that have previously downloaded hotfixes that have been re-signed? Do they have to download again or does their compliance come back to normal?
What is the impact if using WSUS 3.0 SP1, which can't be patched and SCCM 2007?