Yesterday during a discussion I was having with some customers in Taiwan another chat I had with an MVP a month or so ago came back to mind. The question asked was about Independent Software Vendor (ISV, i.e. "not Microsoft") support of Microsoft patches for the OS. Specifically, how long is it reasonable to take an ISV to fully support their product on an OS patched with a particular patch, or rather, update, or a particular service pack?
My response (not the official Microsoft response mind you, but I do not know that there is one) was that as for service packs, the reasonable time frame is "immediately." The fact is that service packs undergo massive beta programs. Every ISV is invited to participate in those beta programs. They can therefore test their products on the service pack at the same time we test the service pack; report bugs to us, and hopefully have a chance to get those bugs fixed before the service pack releases. In other words, I believe there is absolutely no excuse for ISVs to claim that their product is not tested on a particular service pack.
Software updates are different. For updates there is no beta program. ISVs do not get a chance to test the update before it is released. It is reasonable to give them some time to test them before supporting them. The exact time frame should be dependent on the severity of the update, the nature of the systems that should be running the software, and the type of data the ISV processes. For a critical update for a server that needs to process critically important data, such as PII, financial data, etc, 24 hours to 5 days is probably reasonable, depending on the complexity of the update and the ISVs software. For lower severity updates it may take longer.
The discussion got started when the customer found an ISV that specifically would not support any updates since MS04-012, which was released in April 2004. If you want a system updated in the past 18 months apparently you would either have to run an unsupported configuration, or get rid of this software and buy from another vendor.
This is critically important. You have to protect your information assets. You, therefore, have to decide how important they are. If a vendor will not meet your risk management goals, you need to consider a different vendor. It is unfortunate, but that is the only way to protect your network and your information.
Do you agree? Let me know if you have a different way to think of this.
As an ISV, I have to take issue with the comments in this blog. I don't get a chance to see MSXML 4.0 SP2 prior to release, but I do have customers asking me why my application no longer works. After searching in the KB, I find http://support.microsoft.com/default.aspx?scid=kb;en-us;820882
Security Is Tightened When You Post Data by Using the ServerXmlHttp Object. Now i have to code a fix, regression test, package, document, and issue to customers.
Alternatively, I've had issues where, because security hotfixes are cumulative, I get behavior changes in microsoft applications that are unrelated to security - an example is http://support.microsoft.com/default.aspx?scid=kb;en-us;830349.
To some degree, the approach that has been proposed is not altogether different from a black hat releasing a 0 day exploit (perhaps without the malice ;) )
There aren't many good solutions here:
Open beta programs are an option, but they require me to have resources to dedicate for all of my foundation components. (Windows, MDAC, MSXML, SQL, Windows Installer, Office, and the list goes on).
So, it comes down to patch or not to patch - Would one rather be vulnerable until the ISV's complete testing, or have a secure, but inoperable LOB application? That is a very, very hard call to make.
As a practice, we have been implementing MS hotfixes almost blindly, but 2-3 times per year we will have issues as cited above.
As a Manager who supports the software, network, users also has guided work with application development, I wish to say your time schedules look great. However this is not a perfect world, the environments I have worked in or with seem to have the same underlying problems:
1. Wish they had the technical recourses Microsoft has, but don’t.
2. Barely have enough developers to keep up with their existing workload to meet project deadlines let alone deal with a patch update.
3. Do not have a Premier agreement with Microsoft to get the information they need to assist n debugging.
4. Extra test environment for regression testing.
It all really comes down to time, money, resources, all of which most companies are taxed to the extreme in order to reduce company, departmental expenditures.
As a common practice I do not release any Microsoft service pack until two months after its release in order to give software vendors time to release updates and find out where Microsoft made mistakes (because MS does too.), this along with a lot of reading and internal testing to be sure it the service pack does work and all bases are covered. Periodically we still miss something.
As a 'consumer' of software, an admin that handles patch decisions at my office, I'll cut a small ISV some slack. Okay you can argue you don't have the resources but then again, somehow ... I don't have resources either but I can figure out a patch plan and deal with it in my office.
If you are a small ISV I'll let you take some time to deal with the issue...but come on... I'm pretty sure I know the vendors that are referred to in that "we won't let you patch after 04-012 security patch" that was discussed in the blog post because it was told to me and I stuck it on my www.threatcode.com Patching Hall of Shame. That's an accounting software program.... no biggie... I mean only the financial data and what not, that the consultant has to decide whether to accept the risk of patching past 04-012 or stay fully supported by the vendor.
Sorry Mr. ISV...that's an April 2004 security patch for heavens sake... I mean how much time do you need to test for heavens sake.
And if it really does take you that long to test patches and certify them... shouldn't I be concerned how much you are concerned about security in the first place?
That one piece of software also didn't support the SQL sp3 [no biggie ... only protects for SQL slammer you know].
If you can't do patch testing fine...but how about you don't put me in a position to make a risk analysis?
As a mere admin of software.. I beta tested XP sp2. I took resources. It's part of my job as a network admin.
Aren't software companies in the business of writing code? Forgive me... but shouldn't you put resources towards security coding and testing?
I want you guys to stop coding in a manner that you get broken by security patches in the first place. Sorry if that sounds harsh...but if you are getting broken over and over again... forgive me... but maybe you need to revisit why that's happening and take actions to prevent that.
It was once said by a very wise man that the applications broken with XP sp2 where probably broken in the first place.
I was hoping to get a few more comments on this before I posted my OPINION (not an official postion mind you) of the comments. Susan's comments are pretty much spot on though. There is a really important question to be asked if your software keeps breaking when a user patches the system.
That said, I think of this as the cost of doing business. It is also a good partnership. If you are in the business of selling software (or hardware for that matter) then does it not make sense to also provide support for your customers so they can protect themselves while still using your product? That is being a good partner with your customers. There is another side to the partnership. Microsoft cannot possibly test every combination of service packs, operating systems, and third party products. But, we can fix problems when they are discovered before release. After release, fixing problems is much harder. We try really hard not to break things, but sometimes mistakes are made. Those mistakes are much worse when they are not discovered until after release. That is why in the MSDN library we ship a wide selection of beta software - so that ISVs can test their products on it. It may be that not all the betas are in there, and that would be an issue. I'm going to chase that down and see if I can figure out why. Since I was mostly referring to operating system service packs, I did ignore things like msxml 4.0. Sorry about that ommission. I have an e-mail out to the product group on it right now. I will post back when I find out where it is.
Susan's point about apps already being broken is also fair. The fact is that there are certain things that just are not safe, period. When an ISV choses to use a mechanism that is not safe, and Microsoft then decides the mechanism cannot be made safe and blocks it, we have a problem. However, we had a problem from the start with apps using an unsafe mechanism. The RPC Locator service is a perfect example. It is plain unsafe. The EPMappper is safe. At some point, we just have to face up to the fact that to fix problems we have to break things. It sucks, but that is the way it works. ISVs need to be good about helping us do that, and finding out before customers do that their products will break when we fix the problems. Telling customers they have to chose between running an app and being safe is the wrong decision. It did not work for the Ford Pinto, and it should not be the way we do things with software either.
Patches are only one aspect to a multi-layer protection system (hardening, malware protection, network segmentation/filtering, etc...). If you and your vendor have arranged a multi-layer protected environment, give them a break on immediate patch delivery.
For some highly regulated environments, patches must be completely tested after a thorough risk analysis prior to delivery to customers. Depending on the vendor's resources, this can take a while.
Microsoft doesn't make the risk analysis part easy either. Last time we asked Microsoft to provide more detail on the patches to help us in performing risk analysis, their lawyers in the room somehow prevented a successful answer. They did offer beta patch testing, however. I'm not sure I can blame then when looking at the overall risk situation (the risk of more reverse engineered exploit code arriving faster vs. one industry segment complaining). However, if the regulatory bodies start to get involved (and they are), this transparency might get a little clearer.
Steve, that may be the most insightful comment so far. Absolutely, if you have a way to obviate the need for a patch, then holding off patching is the right way to do it.
The problem with service packs is that it is often very difficult to gauge whether you do or not. However, a solid defense in depth strategy is critical to successful protection, even in the absence of patches.
The part about Microsoft not being able to give more detail on patches is a comment I hear about once a week or so. I know. I have tried for years to get more information on them, particularly on what testing was done, and keep running into legal arguments why we can't. Sorry. Can't help you more there.
I just wanted to follow up on the MSXML service pack issue. MSXML ships mostly with other products, notably Windows and/or SQL Server/Visual Studio. Generally speaking MSXML service pack betas will be available as part of those products. There is also a lot of MSXML information on the MSXML team blog: