Roger's Security Blog : Patch Management

The Microsoft Security Update Guide

rhalbh — Thu, 13 Aug 2009 15:30:23 GMT

I know that these news are not new but I was away when we announced it and to me it is important enough to take it up afterwards.

Over the last few months we worked on a document explaining everything which is going on around an Update Tuesday. So, what is an Advanced Notification, what information do you find an a Security Bulletin and how should you handle this kind of information etc. We announced this document during Blackhat. To quote from the download page:

This Guide was designed to help IT professionals better understand and use Microsoft security release information, processes, communications, and tools. Our goal is to help IT professionals manage organizational risk and develop a repeatable, effective deployment mechanism for security updates. In this Guide, you will find a convenient glossary of terms, an overview of the Microsoft Security Bulletin process, and a stage-by-stage review of Microsoft Security Updates.

I think Michael Grady did an outstanding job pulling this all together.

It can be found and downloaded here: The Microsoft Security Update Guide

Roger

Patch Management, a key step towards compliance!

rhalbh — Fri, 22 May 2009 15:18:34 GMT

As you might have read, I recently blogged about my infrastructure and the future of a platform towards a better management of compliance – honestly, I actually played with our latest technology .

I wrote about

Now, a necessary and very important next step towards compliance as well as a secure environment is a sound Patch Management process and then – in the second place - the underlying technology. I blogged several times already about Patch Management as I see a lot of companies failing to deliver on this. I recently wrote a post called Patch Management – Cover the whole 9 yards. in there I mention different papers you could/should read:

and I reference Christopher Budd’s Ten Principles of Patch Management:

Service packs should form the foundation of your patch management strategy
Make Product Support Lifecycle a key element in your strategy
Perform risk assessment using the Severity Rating System as a starting point
Use mitigating factors to determine applicability and priority
Only use workarounds in conjunction with deployment
Issues with Security Updates are documented in the Security Bulletin Master Knowledge Base Article
Test updates before deployment
Contact Microsoft Customer Support Services if you encounter problems in testing or deployment
Use only methods and information recommended for detection and deployment
The Security Bulletin is always authoritative

First of all (and you see that in the articles referenced above) it is of outmost importance to have a process in place. Basically the core schema to run such a process is:

I have seen different complexities to deploy such processes. From highly complex to pretty simply and straight-forward ones. The ones of you who know me know, that my preference is KISS (Keep it Simple, Stupid). So, make the process as complex as necessary and as slim as possible.

So, once you have the process in place and take a conscious decision, the question is about deployment and reporting.

So, let’s talk about technology now.

In order to get an overview over the state of your computers, you might use the Microsoft Baseline Security Analyzer. This is an excellent tool to scan your Windows machines and get an overview of the security state of the machines. It might not deliver the same level of sophistication as very expensive tools, but the difference is: We provide it for free and – in my opinion – it gives you a good starting point to look at vulnerabilities including the level of Security Updates of a given PCs. Here is an example of one of these assessments:

But this does not really resolve your base problem about the Security Update compliance of the computers on your network as well as the distribution of them. From my point of view, there are different options to do so:

If you are a small and medium business, one of the coolest solutions for you to go is System Center Essentials. It is System Center Configuration Manager, System Center Operations Manager and Windows Server Update Services in one package. However, it is limited to 30 servers and 500 clients. If you are in this limit, it rocks.
System Center Configuration Manager: If you already use this technology to distribute software and configurations, leverage this.
Windows Server Update Services: It is kind of unbelievable but this is free! So, to be clear – we do not charge for it! You can download and install it and it scales even for large Enterprises (did I tell you already that it is free ?).
A third-party solution

I am using WSUS and am more than happy with it. The way I am organized is, that I get regularly a mail from WSUS with the current state of “the nation”:

As I am mail-driven, this allows me to see, what I have to do with regards to WSUS. I then can log-on to my WSUS server to get more granular reports:

From here on, I can decide, which actions I want to take, based on detailed reports I can get by clicking one of the texts in the UI:

BTW: this machine is patched in the meantime – so do not even think about it

Even if you cannot enforce the security update level technically that way (and we will talk about Network Access Protection in a later post), it at least helps you to understand, where you stand and what you have to do in order to get compliant.

Again (as I did so often) my call to action to you: Make sure that you have a straight-forward process in place and then use technology (like WSUS) to deploy the updates and ensure that you have deployed them correctly!

Roger

MS09-017: An out-of-the-ordinary PowerPoint security update

rhalbh — Wed, 13 May 2009 16:10:08 GMT

Our Security Research and Defense team blogged on the PowerPoint security update we published on Tuesday. There are a few things which were not “business as usual”:

The update for the Windows version of PowerPoint went out before the Mac version. The reason is that we did not want to hold the Windows-version which could protect a big majority of our customers
We removed support for the PowerPoint 4 converter to reduce the attack surface significantly
We addressed 14 (!) vulnerabilities in PowerPoint (do not tell me that we are not transparent with these things) – only one is publically attacked at the moment.

Read the details yourself: MS09-017: An out-of-the-ordinary PowerPoint security update

Roger

Patch Management – Cover the whole 9 yards

rhalbh — Thu, 26 Mar 2009 22:04:45 GMT

I pretty often have discussions about Patch Management with our customers. I think it is a very important discussion as I see too many customers not patching at all.

However, taking the shining examples – they often look at the Microsoft product suite “only”. You might remember that I blogged about my experience with this on my home PCs: 98% unpatched – and I am one of them :(

Now, this transfers to the enterprise business as well. If you look at our latest Security Intelligence Report, we have an interesting chart to show you the whole problem:

This chart shows the Microsoft share of the industry-wide vulnerability disclosures. What I want to show you with this chart is that our share of vulnerabilities in 1H 2008 is below 3%, which means for you if you are implementing a patch management strategy, you have to make sure that you cover the other 97% of vulnerabilities as well.

I am well aware of the fact that this does not show your risk distribution. Based on your usage of our technology as well as the fact that criminals use our platform more for attacks as there is more to gain because of the wide distribution, your risk profile will be distributed differently. However, there is no discussion that you need to cover all the products you have in place.

The actual reason, why I write this post are two articles I read today, which show perfectly what can happen if you omit the rest of your environment – including your hardware:

On our website there are several good resources with regards to patch management:

Conficker showed us again that a sound patch management process is the foundation for your defense/security/risk management strategy. So, please if you did not yet deploy security updates – please go ahead and start. The earlier the better and base it on the principles of patch management referenced above:

Service packs should form the foundation of your patch management strategy
Make Product Support Lifecycle a key element in your strategy
Perform risk assessment using the Severity Rating System as a starting point
Use mitigating factors to determine applicability and priority
Only use workarounds in conjunction with deployment
Issues with Security Updates are documented in the Security Bulletin Master Knowledge Base Article
Test updates before deployment
Contact Microsoft Customer Support Services if you encounter problems in testing or deployment
Use only methods and information recommended for detection and deployment
The Security Bulletin is always authoritative

Roger

Qtel’s Guide to a Faster Internet Experience

rhalbh — Mon, 09 Mar 2009 11:33:23 GMT

I like that: As you probably know, I did a tour through the Gulf when we launched the Security Intelligence Report last year. One of the reasons was that we know that the Gulf has a pretty high malware infection rate. You can read this in the corresponding blog post: Security Intelligence Report v5 Live!

Now, QTEL (the ISP in Qatar) released an interesting document called Qtel’s Guide to a Faster Internet Experience. What I like about it is that most of it is about security but it actually addresses the user where it “really hurts”: Internet performance.

You can read it yourself at the link above

Roger

Would a properly managed IT have withstood Conficker?

rhalbh — Wed, 04 Mar 2009 17:37:09 GMT

Before I start here: Let’s be clear that I will not say (and will never say) that if a customer was infected with Conficker he had a poorly managed network!

I had a lot of discussions over the course of time about the reasons for customers being infected. We all know the attack vectors of Conficker but what are the real reasons behind it?

Poor or no Patch Management: We are coming back to my Russian Roulette post back in January. If you decide not to patch or leave it to the admin to decide, in my opinion this is negligent. And now, please, do not tell me that this is a Microsoft-only problem. Base don the Security Intelligence Report, we are responsible for 3% of the industry-wide vulnerabilities. So, do not forget to patch the other 97% as well!
Unmanaged Machines: This comes down to compliance management as well and is not an easy problem to be solved. But we have seen customers who thought that they were fully patched just to find some unmanaged machines which were not – and these machines were the starting point of the infection. Let’s be clear on this one: This is a problem which can be addressed but it is a project you have to run with the corresponding investment. There is technology out there like 802.1x to keep them off the network or IPSec Auth to make sure they are not able to talk with your key machines – but you have to deploy them.
Weak (or inexistent) passwords: Once you had (or have) Conficker on the network, weak passwords are often a pretty good vector for Conficker to spread. Again, it is about compliance management.
Everybody is an Admin: We can now debate about this again, who is to blame with this. It is a fact that a lot of users run as Admins (yes, in Enterprises as well) because certain applications do not run without Admin privileges. The virtualization part of Windows Vista definitely helped to reduce this. Since Windows Vista (and Windows 7) I am not running as Admin anymore nor does anybody else in my network! This is once of the key achievements of UAC.
Unsupported Operating Systems: It is really unbelievable how many NT4 we still find out there. We retired NT4 SP6a on 31.December 2004. Please do not come now and blame us for our policy. Today we support our products for 10 years at the supported service pack level (Business and Developer products). I know that there are reasons why you cannot upgrade – but there are a lot of machines out there which could be upgraded! Additionally it is sometimes worrisome to me how often I see old Operating Systems and unsupported applications being connected to the network without having any further protection and/or shielding.
Anti-Malware Protection: This is a very difficult area now. There are customers who had Antimalware in place and did their best to having the signatures updated – however the AV-vendor failed to protect their customer base. We have been out with a signature for Conficker.B since December 29th – not only detecting Conficker.B but removing it. And there are some vendors talking very loud about Conficker did a very doubtable job.

I know that it is not easy but looking at the reasons above, I am convinced that a well-managed environment would have had a good chance to withstand Conficker. Well-managed meaning:

Having proper policies in place where Business Risk Management is being seen as a fundamental part of an IT management
These policies are enforced through administrative measures and audits as well as through technical means (yes, the auditor can be your friend).
Violation of policy will be punished.
Apply not “best practices” but good practices to your environment

It just showed us once more that running a network of a certain size is an engineering practice and not an art. Today’s economical situation does not help here either as a lot of companies want to save cost. However, a well-managed network to me is an inexpensive network as well – and a secure network! So, we should definitely think about this further. I am convinced that in today’s time we have to move from “best of breed” to “best of need” and in addition we have to make sure we deliver and you deploy a “best of need integrated platform” to address the challenges outlined above. So that you can concentrate on a business strategy as well as on processes!

This raises the 1 Million Dollar question: How the hell can you make sure that you know what you need. Well, you have to do Business Risk Management. A lot of companies – to me – miss the “business” in the statement above it they do Risk Management at all. From my point of view, it is not the CSO’s job to decide about the risks acceptable for a company. It is the Management Board. At the end of the day it is a business decision and not an IT decision! However, it is the CSO’s job to make sure the Management Board understands the risks they are taking on a level which is understandable for a business leader.

Let me add one final statement. Microsoft IT has a pretty tough job to do with all these geeks connected to the network running all sorts of beta software. However, I did not feel any disruption from Conficker. So, there is a good chance that they did an excellent job to keep it out. So, it is doable.

I will try to blog more about the platform in the near future. I started to bring these pieces together in my test environment to get some hands-on experience and I want to share more of this with you

Roger

Is there a Correlation between Stolen Software (Piracy) and Security/Patching?

rhalbh — Tue, 20 Jan 2009 23:22:00 GMT

Remark: A few weeks ago I made a post where I asked you about the correlation between Piracy and Security. I was talking about Piracy (stolen software) and got a lot of answers about Privacy (Data Protection) . So the following post is about stolen and illegal software…

I was recently asked in a panel whether there is a correlation between piracy rates and malware infections in a given country. I am convinced that this is the case in the consumer space because I suspect many pirated copies are not protected. But can I prove it?

You might have seen it: We recently filed some cases regarding piracy in different countries. These cases go after software resellers who allegedly violated Microsoft’s copyrights and/or trademarks by illegally selling counterfeit software and software components via online auction sites – which is a serious kind of fraud.

But where I would really like to understand more is when it comes to the relationship between Piracy and Security/Patching. To me, there are different “types of piracy”, which might have different impact on security:

Criminals that steal software and then sell it. From my personal experience the end-user is often unaware of the fact that he/she is running non-genuine software. So, there is a good chance that Automatic Update is switched on
People downloading pirated copies of software from peer-to-peer networks or other sources. Here the problem is different as these people most probably do not have any patch management solution switched on.

To be clear: Some time ago, we decided to deliver critical security updates via Automatic Update to non-genuine versions of our products. This is not to protect the thieves but to protect the ecosystem. I often get push-back that this is not true, so let me clarify.If you go to the download center or Microsoft Update you will not be able to access these sites with pirated copies but switching on Automatic Update will allow you to get the critical Security Updates.

The reason why I am telling you this is because I would like to do some statistical exercises with you. There is data on Malware Infection Rates in our Security Intelligence Report. This data is compiled from results of the Malicious Software Removal Tool which is mainly delivered through Microsoft Update and Automatic Update. So, we will see mainly machines that are getting regular updates.

If we look at the countries in EMEA, this is the extract from the report which shows the 15 countries in which we found the most infections and the 15 in which we found the least in H1 2008 (the number is the number of infections we find per 1000 executions of MSRT):

Country	Rate
Algeria	19.5
Libya	19.5
Portugal	19.6
Yemen	20.1
Lebanon	20.2
Macedonia	21.1
Jordan	21.6
Tunisia	21.9
Turkey	21.9
Saudi Arabia	22.3
Egypt	22.5
Iraq	23.6
Albania	25.4
Morocco	27.8
Bahrain	29.2
Afghanistan	76.4

Country	Rate
Rwanda	4.2
Austria	5.2
Germany	5.3
Finland	5.7
Latvia	6.3
Denmark	6.8
Switzerland	6.9
Czech Republic	7.1
Italy	7.1
Ireland	7.3
Belarus	7.6
Sweden	7.6
Netherlands	7.8
Nigeria	8.2
Poland	8.3
Norway	8.3

So, this is about malware.

Let’s look at at Piracy figures now, using figures from a report by the Business Software Alliance. So, let’s do the same and look at the 15 worst and 15 best countries in terms of piracy (these are 2007 figures):

Country	Piracy
Albania	78%
Kazakhstan	79%
Côte d'Ivoire	81%
Kenya	81%
Nigeria	82%
Montenegro	83%
Ukraine	83%
Algeria	84%
Pakistan	84%
Iraq	85%
Libya	88%
Yemen	89%
Azerbaijan	92%
Moldova	92%
Armenia	93%

Country	Piracy
Luxembourg	21%
Austria	25%
Belgium	25%
Denmark	25%
Finland	25%
Sweden	25%
Switzerland	25%
Netherlands	28%
Norway	29%
Israel	32%
South Africa	34%
Ireland	34%
UAE	35%
Czech Republic	39%
Hungary	42%

So, what does this tell us? Well, nothing really yet. So, from here, what we could do is looking at the rankings. (Being an engineer, I love to play with figures :-))

I started to compare the rankings of the different countries and tried to understand the difference in the relative ranking between Piracy and Malware Infection Rate. Let me give you an example: Switzerland ranks 5^th lowest on Malware and 2nd lowest on Piracy. So, the difference there is 3. Ukraine, on the other side ranks 22nd on Malware but 51st on Piracy – so, there is a difference of 29 which is significant. So, they are doing about average when it comes to the malware infections but really bad in Piracy (actually in Ukraine 83% of all software is not genuine).

If we draw a graph with these differences it shows a clearer picture than the tables above:

So this tells us that most of the countries just rank about 5 places apart between Malware and Piracy!

Even though we are only covering PCs with the Malicious Software Removal Tool running in the malware infection rate, most countries that are bad/good on infection rate are bad/good on piracy.

But with this statement, this would lead us to the next question: Why is this the case? There might be different reasons for that:

We know that Peer-to-Peer networks are a source for malware. So there is a good chance that people who deliberately steal software have it on Peer-to-Peer networks, or other untrustworthy sources, and get the malware from there.
People who pirate software are careless anyway and do not run Anti-Malware software, or have it but do not update it
People who pirate software do not patch their PCs because, in their mind, they think that running Microsoft Update or any other update mechanism will lead to them being caught. This would be interesting to investigate further but unfortunately I have no data I can make public on Microsoft Update hit rates in the countries above.

To make one point clear: The statements above are mere speculation. Today I have not enough intelligence available in order to strengthen one of the points above. On the other hand I think I have shown that there might be a correlation between Piracy and Security and I would guess it would be easier to convince consumers to patch their machines (and therefore get basic protection) if they run genuine copies rather than stolen copies!

Roger

Russian Roulette with your Network

rhalbh — Sun, 04 Jan 2009 15:17:00 GMT

First of all, before I really start, I hope that you all had a great start in 2009. Mine was actually pretty mixed. The good side was, how my year really started and what I saw when I looked out the window at January 1st (yes, I was on vacation skiing and this was how the view was almost each and every morning):

But honestly, this is not the only reason, why I wrote this post. There is another one which is much, much more serious:

Unfortunately there are still plenty of customers playing Russian Roulette with their network. This term was actually used by one of our security engineers – who was kind of upset to say the least – who had to work December 31st and January 1st because of customers still not having rolled out MS08-067 – and not just one! We ran to our limits with regards to support capacity in EMEA.

Just to remind you: This is the Out of Band security update we released back on October 23rd and which then was pretty soon attacked by Conficker.A. But it seems that a lot of customer did not care back then – they were not attacked, so why bother? In the last days of 2008 Conficker.B broke out and even though it was not spread too widely, the customers who were hit (or still are hit) are hit very, very badly. Account Lockouts all over the place, admin passwords that were grabbed (often the Domain Admins) etc – and we had some really upset engineers as they had to work instead of having off because some customers were not up to their duty (and this is what it is for me!).

And this is not the end of the story:

For quite a while, our Anti-Malware solution was the only one, which was able to remove the thing. And without an Anti-Malware solution it is close to impossible to actually get rid of it. As always, all the information about the malware was shared amongst VIA (Virus Information Alliance) to all the partners.
NT got infected as well and the calls came: What shall we do now? Well, there is not too much you can do. As you might know, Windows NT is out of support for a long time (since December 31st, 2004 - see our Lifecycle Page if you need more information). Isolate your Windows NT boxes (as you should have done a long time ago) and migrate away from it. I know that there are still a lot of machines with NT embedded – isolate them and work with the vendors to get to an up to date version of the OS.

Let me add a final comment: The story above is not a Microsoft-only story. The same processes and technologies around patch management have to be applied to each and every component of your environment. Back after the Blaster times, we start to tell the consumer to apply three things to their PC to protect it:

Switch on your Firewall
Keep your Software Updated
Run an Anti-Malware software and keep it updated

Guess what: If you would have applied 2 and 3 to your network, you would not have been hit by this problem.

Roger

98% unpatched – and I am one of them :(

rhalbh — Fri, 05 Dec 2008 17:57:09 GMT

Well, you saw my post earlier this week on the 1.96% of PCs being updated according to Secuina. Well, as time does, I decided to install this tool as well to look at it. I did an initial scan on my home PC and this was the outcome:

Outch, this hurts my soul but shows as well the problem: I definitely have all our software updated and with must of the solutions above, I have the updates switched on (except Apple, where I switched it off when they wanted to install Safari as an update :()

But honestly, the tool is pretty cool. If you switch to advanced mode, you even get pretty detailed information:

So, this makes me really think. This is a PC which I really look after and keep it updated. Nevertheless I seem to have failed.

This shows me the fundamental problem: If I am not able to keep it up to date, how shall my Mom and Dad? The Secunia Personal Software Inspector helps a little bit but I am nut sure whether my parents are able to handle it. So, what we are basically missing is a central point and mechanism to distribute security updates. But who controls this channel? Who ensures that no criminal can get access to it? That no viruses are distributed?

Still a long way to go…

Roger

P.S: Do not even try to attack my PC based on these vulns – they are closed in the meantime

Only 1.91% of PCs are patched!

rhalbh — Thu, 04 Dec 2008 19:59:49 GMT

Well, honestly, I am not completely clear how statistically relevant this data point is. I just read it in a secunia blog where they published figures of users of their free solution. This is data of the last few weeks and looks into the results of the first scan of the product on a PCs. It covers 20’000 users. Not being a statistician I cannot judge the quality of the results but even if the situation is 10 times better it is still very, very, very bad. This is the table they published:

Number of insecure programs per PC/user:

0 Insecure Programs: 1.91% of PCs

1-5 Insecure Programs: 30.27% of PCs

6-10 Insecure Programs: 25.07% of PCs

11+ Insecure Programs: 45.76% of PCs

By "insecure program" it is understood, that there is a newer version of the program available from the vendor that corrects one or more vulnerabilities, but the user have yet to install the secure version. A vulnerability in a program can be exploited by hackers to anything from compromising a PC, to automatically install trojans/viruses, to sniff out private information (passwords, credit cards information, etc.).

If I would extrapolate that to the situation with MS08-067… no, I do not think that I wanna do that :(

Remember – this is not only Microsoft, this is everything they have on the PC

Roger

Attacks on MS08-067

rhalbh — Wed, 26 Nov 2008 16:09:00 GMT

As we were pushing on our Out-of-Band release earlier this month we tried to make you understand that immediate deployment is needed as the vulnerability is high risk. Otherwise we would not have gone out of band…

Interestingly enough, we have not seen widespread attacks since now. Earlier today now we released different pieces of information on the two key blogs on that:

Microsoft Security Response Center: MS08-067 Update: November 25
Microsoft Malware Protection Center: More MS08-067 Exploits

The reason why I post and why this attacks makes me a little bit nervous is that I hear from too many customers still that they did not yet deploy and the reason behind is that “they heard that we might have issues with this update”. Sorry, this is blank nonsense.

To be clear: Out of all support cases Microsoft has received regarding MS08-067, all of them (and I mean all – no exception) turned out to be caused by another issue and/or mis-configuration and not MS08-067! So, there were no issues with this update so far.

It is your choice now to decide whom you base your risk assessment on: On some web pages telling you that they heard or on us.

Whatever you do, base your risk assessment on the fact that there is somebody out there exploiting the vulnerability

Roger

Servers still not patched

rhalbh — Fri, 29 Aug 2008 13:20:00 GMT

I just read an article this morning on Linux servers under the Phalanx gun: A problem with people, not code. There were quite some things which made me think when I read it:

There was a statement in there, which I – obviously – did not like at all: Linux may be inherently more secure as a system, which is always an interesting discussion. The guy writing the blog post claims that Linux is easier to secure than Windows, which I completely disagree with. If you know what you do you can secure each and every system. However, we do a great deal of work to make sure that our systems are as secure as possible by default and additional provide you with tools (like the Security Configuration Wizard) to make sure you can secure the system as far as possible and additionally run as secure as possible. We know and proved it with a lot of figures that our systems have by far less vulnerabilities than others (e.g. http://blogs.technet.com/security/archive/2008/05/15/q1-2008-client-os-vulnerability-scorecard.aspx) and third-party research showed clearly that our systems are less at risk than others.

But as I commented several times already, this discussion does not really lead to more secure systems but just some entertainment for people who like these debates.

Coming back to the article above: One of the conclusions in the article is, that patching is often a people and process problem, rather than a technology problem. This is not new either. The question to me is, why do people not deploy? We do customer surveys about their satisfaction with Microsoft every now and then. People are still not too satisfied with the security of our products. So, there is still a lot of work to do. However, if we ask then whether our updates are easy to deploy, we get a very, very high rating all across the segments and audiences. So, why do they not deploy? Is it because they are afraid of the downtime? Could be, so we have to work harder to reduce the number of reboots (is this different in other OS? I do not know but I doubt). Is it the tools? Is it lack of knowledge? Is it ignorance?

I do not know but would love to understand

Roger