Ramblings from another nerd on the grid
I'm the people part of the equation. Recently the process and technology ran right over me and turned me into IT road kill. So what happened? Could it have been avoided? Definitely...
Let's set this up by describing my typical work environment. Like many field employees, I am a mobile worker. When I am traveling I use a variety of internet connections. When I am home, I use my broadband isp connection. I am in the office about 3-5% of the time so it's rare to have a corporate connection.
Know Your Users
When designing a patch management strategy for mobile workers, where would you have them get software updates? What technology do you use?
In the case of the Microsoft IT org, they use SMS for inventory, patch and software push/pull distribution. With the new version of Windows Software Update Server (WSUS) v3 getting ready to ship, I guess they were anxious to start using it. However, they got a little ahead of themselves and it tripped me and a few other folks up.
In early March, I noticed my secondary work laptop stopped getting updates from windowsupdate.microsoft.com. It is running Windows Vista Enterprise and is joined to one of our corporate domains. As such, it must adhere to the corporate desktop standards for anti virus, malware prevention, etc. Some of those standards are implemented via Group Policies that flow from our Active Directory (AD) domain controllers. One of the group policies I received updated my Windows Update client to point to the internal WSUS servers.
Although the initial problem is probably apparent at this point, it's actually worse than that. You see, I received that GPO update, but I didn't really see the issue until much later. By the time I noticed I wasn't getting updates (see screen shot above) from the public update site, it was too late. Our VPN quarantine process already had several patches that were required to pass quarantine but guess what, I can't get them because they are on the internal WSUS servers. That's a catch 22. I'm screwed.
Or am I?
March was a super busy month for me as you know. We wrapped the Vista launch events, I developed the Longhorn content for Q4 and then headed to Orlando to deliver sessions at TechMentor and Windows Connections. However, while there, I saw Chris Henley hit the same issues with his laptops. That put a fresh perspective on the issue and I knew it wasn't just me. Time to start hacking into this.
So when I got home, I started sand boxing the problem. Good trouble shooters learn to draw boxes around the problems and isolate components. Excluding components makes it easier to find the faulty component. The error message displayed doesn't tell much so you have to "Get Help". That ends up taking you to some help that advises you to make sure you aren't blocking the windowsupdate servers with a firewall, and to also add their URL's into the browser safe sites list. Seems reasonable. This starts to clue you in it's a destination unreachable condition. We just don't tell you what destination.
To get the destination, you have to dig into the Windows Update client log. Not many users are going to go down that path. But I'm a hacker like you so I head on over to c:\windows\WindowsUpdate.log. I see a bunch of log entries pointing to Microsoft internal IP addresses. I smell a rat. I now know what is going on. So I dig a little further. Sure enough, the group policy update set the following registry entries:
Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate]"WUServer"="WUStatusServer"= [HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU]"UseWUServer"=dword:00000001
Windows Registry Editor Version 5.00
I have removed the ip addresses and ports to protect the guilty. Ok, now I know exactly what is going on. My Windows Update client is pointed at some internal servers for updates and I can't get updates. So I deleted the registry entries and grabbed my updates from windowsupdate.microsoft.com. I know, that's cheating. But hold on, it gets better.
Bug or "By Design" ???
So being the good corporate citizen I am, I report the issue first to our internal Windows Vista discussion alias. I don't know if what I am seeing is actually a bug yet or not. I suspected it's just a case of the MSIT org not understanding how field employees work and under estimating the impact of re-homing the update client to an internal WSUS server as opposed to using the public cluster.
In the ensuing discussion on the alias, I find out that some new WSUS clients are being pushed out. Sure enough, I receive the WSUS v3 client (see the screen shot above). Notice anything different?
Windows Update (WU) Client v3
If you notice at the bottom of the text, there is now a link that you can use to pick up updates from the public Microsoft Update Service. Very nice. It would have been nice to have this client pushed to my machine back in early March before the GPO updated me to point to the internal servers.
See what I mean? This wasn't a technology issue at all. This was a people and process issue. This happens all too frequently. Fortunately, it doesn't happen often in Microsoft. So what's next on this issue?
The End Game
Well, from the technology perspective I would have preferred the WU client to time out trying the internal servers then fail over to the Microsoft Update Service. This is what my anti virus tool does. It's tries the list of servers and they can be a mix of locations and protocols (ftp, smb, http). Maybe we'll see that in the next versions of the WU Client.
So the moral of this story is to make sure you aren't creating end user IT road kill. Put on their hat and assess their needs fully. In our case, we might need to assess creating a mobile user AD domain, OU, or backing this change back out. I think our helpdesk folks are just going to get too many phone calls on update failures.
Happy Easter !!!
[UPDATE] The Windows Software Update Server (WSUS) registry settings are documented at http://support.microsoft.com/kb/328010.
WSUS fail-over to a public server assumes that the same patches are released on both.
Many organizations might not want all MS releases available to end users (e.g. internal testing finds a patch kills an application), therefore letting them lose on the public server via fail-over defeats this.
Of course, working for MS you don't have this problem ;)
Yea, I know. That's the dilemma. WSUS is great for patch testing and approval. Where I got caught was the VPN quarantine check and the GPO update.
Since I was behind on updates, VPN quarantine said I could not pass go and connect to corpnet. And the GPO said I could only connect to a corpnet update server.
Of course, this would not have happened if Windows Server "Longhorn" Network Access Protection (NAP) was fully deployed. If NAP was controlling the network policies, then it would have spotted my machine was not compliant then send me over to a group of remediation servers. Since NAP isn't fully rolled out yet, I didn't get that benefit.
We ran in to the similar problem with people using company Windows Vista notebooks while working at customers for longer periods of time with no possibility to use VPN: they would always get this error for not being able to contact our internal WSUS server. For this reason we are considering publishing our WSUS server through ISA Server on a public URL and specifying that URL in the Group Policy (internally redirecting it to the internal server using split DNS), so that we can still push updates to those users.
Another problem was that until recently the Windows Vista Ultimate Extra's were not published on WSUS, which meant that some users running this OS edition would remove our internal server from the registry keys so they could get them from the Microsoft Update Service.
This upcoming change in the Windows Vista sort of brings us back to the situation in Windows XP where users could use IE to browse to Windows Update or Microsoft Update and download and install updates that we had not approved in WSUS (yet). Afterall, part of the point of using WSUS is that the company IT department can control what gets installed on our notebooks. I sure do hope that there will be a Group Policy setting to disable the "Check online for updates for Microsoft Update Service" option, or that we can configure that it should only allow this when the user is not on the domain, and that we can configure what kind of updates can be installed this way (so similar to how we have configured WSUS to auto-approve Security and Definition Updates, we may want users to only be able to those two specific types of updates from Microsoft Update Service, and not for instance, drivers and service packs).
Control is key in many environments. I agree it's prudent to test new updates and verify they don't break other mission critical apps.
Remediation is key. Publishing part of your WSUS server environment is one solution. NAP will be another.
I'll check to see what GPO blocks are available in WinxP and Windows Vista to prevent getting updates from the public download catalogs and update.microsoft.com servers.
As you'll recall, I posted some information about the Microsoft IT organizations implementation of WSUS