I was recently caught in a tricky problem: The clock of one of my host servers ran out of sync.. – significantly. The core problem was that my Mediacenter (which is domain integrated) started to record about 6-8 minutes too late but this is not the reason why I post.

The actual reason was that I tried to resolve this: My DCs are virtualized – one on a Hyper-V server and one on a Virtual Server. As both have the corresponding add-ins installed, by default the guest synchronizes the time with the host. If the host clock is now not accurate anymore, this is transferred to the guest (which is a DC and which then synchronizes this across the whole infrastructure). As this happens slowly, I did not realize this until my Mediacenter did not capture the whole news anymore…

Now I checked the time server settings of my DC and it synchronizes its clock with time.windows.com and NTP is open for the DC – therefore the synchronization is successful, resets the clock to the right time and then the Hyper-V Integration Services kick in and set the clock back to the time of the host (which is wrong) and the wrong time is again synchronized across the network . (I hope this was now confusing enough)

What I did now – and what I would suggest that you do that (at least with the knowledge I have today) – is disabling the time synchronization between host and guest at least for DCs as they update their time from the time server as described above. Since then, my time is correct again.

Roger

P.S. As you know – I am Swiss. And one of the worst thing which could happen to a Swiss is an incorrect watch

I pretty often have discussions about Patch Management with our customers. I think it is a very important discussion as I see too many customers not patching at all.

However, taking the shining examples – they often look at the Microsoft product suite “only”. You might remember that I blogged about my experience with this on my home PCs: 98% unpatched – and I am one of them :(

Now, this transfers to the enterprise business as well. If you look at our latest Security Intelligence Report, we have an interesting chart to show you the whole problem:

This chart shows the Microsoft share of the industry-wide vulnerability disclosures. What I want to show you with this chart is that our share of vulnerabilities in 1H 2008 is below 3%, which means for you if you are implementing a patch management strategy, you have to make sure that you cover the other 97% of vulnerabilities as well.

I am well aware of the fact that this does not show your risk distribution. Based on your usage of our technology as well as the fact that criminals use our platform more for attacks as there is more to gain because of the wide distribution, your risk profile will be distributed differently. However, there is no discussion that you need to cover all the products you have in place.

The actual reason, why I write this post are two articles I read today, which show perfectly what can happen if you omit the rest of your environment – including your hardware:

• Hacking The Router Patching Conundrum
• New worm can infect home modem/routers

On our website there are several good resources with regards to patch management:

• Ten Principles of Microsoft Patch Management
• Update Management
• Update Management Process

Conficker showed us again that a sound patch management process is the foundation for your defense/security/risk management strategy. So, please if you did not yet deploy security updates – please go ahead and start. The earlier the better and base it on the principles of patch management referenced above:

1. Service packs should form the foundation of your patch management strategy
2. Make Product Support Lifecycle a key element in your strategy
3. Perform risk assessment using the Severity Rating System as a starting point
4. Use mitigating factors to determine applicability and priority
5. Only use workarounds in conjunction with deployment
6. Issues with Security Updates are documented in the Security Bulletin Master Knowledge Base Article
7. Test updates before deployment
8. Contact Microsoft Customer Support Services if you encounter problems in testing or deployment
9. Use only methods and information recommended for detection and deployment
10. The Security Bulletin is always authoritative

Roger

Recently I decided to spend some time to implement some new technologies in my environment at home. The environment itself is a mixture between test and production. If you are reading this post on www.halbheer.info/security, you are already accessing this environment. So, I host my web server, mail server etc. there, all our private mails are received there but I am still trying to deploy beta-technology as I want to understand the challenges you all will go through when you run these products in your production environment – being well aware that 8 or 9 servers and a few clients is by far not comparable with what you do out there.

Now, I decided to write a few blog post about how I integrated our technology as I wanted to prepare the environment for the active protection technologies being part of our next generation of the Forefront suite called Stirling as well as some other cool stuff we recently released (like NAP) but I never had actively in my hands. I decided to share some of the experiences and challenges with you as I went through this (it was a lot of fun for me).

Let’s start with PKI first – which I deployed a few years ago already. Even though I know that there are quite some companies that are still refraining from deploying PKI, I am definitely convinced that over short- or mid-term there is no way around it. Certificates and the authentication linked to it is already all across an infrastructure. So, let’s start there.

Before I joined Microsoft, I was working at PricewaterhouseCoopers running PKI projects mainly with regards to policy development, processes and organizational concepts. These projects (not only our part but including the software licenses) tended to be huge and very time- and money-intense. One of the reasons for that was, that it was far away from being commodity. Believe me or not but back then (this was around the year 2000) I was saying the PKI cannot take off before Microsoft integrates it into the client.

I then moved to Microsoft and we released XP with already a pretty good PKI integration, especially when we added the Windows Server 2003 PKI. However, there was a downside: Too many customers just went through the wizard and installed a PKI – and generated a problem. Even though today I am convinced that you need not these thick books of paper before (I am not paid by the number of pages I write ) you need to do some planning. Especially you have to make sure you understand the application of the PKI before you deploy it as well as the assets you are protecting. This means planning, this means concepts, this means experience.

There are actually some pretty good papers on our website which can help you there:

• Designing a Public Key Infrastructure
• Best Practices for Implementing a Microsoft Windows Server 2003 Public Key Infrastructure
• Windows Server 2003 PKI Operations Guide
• Troubleshooting PKI Problems on Windows Vista

These documents help you to understand, which decisions you need to take before you start to deploy. Decisions, such as (not complete):

• Enrollment processes
• Protection of the different private keys
• Certificate Lifecycles (yes, there are different validity periods and they will depend on each other)
• Revocation and distribution of the revocation information
• etc.

So, I took it pretty straight-forward: I decided that I needed a PKI for various purposes but definitely no high-trust certificates (I did not have a HSM – a hardware device for the protection of the CA’s private key – anyway). So I looked into naming of the PKI, lifetime of the root cert etc. and then went for it.

So, I installed initially a Root CA, which directly issues certificates for machines and websites. I even put it on a DC. The reason for that was two-fold:

1. I definitely did not need higher security for my CA than the security level of the DC
2. I did not have more server available

Well, honestly, I started to deploy certs and later on re-started again… I stupidly named my PKI “Root” and became the joke of my friends at Microsoft. When you looked at my “Trusted Certification Authorities” on any of my computers, there was one "called “Root”, which is really descriptive . So, I wanted to get rid of this problem and re-installed the stuff (which you probably do not want to do – therefore think first not like me). I even have the root cert publically available as I need it from time to time outside my infrastructure as I am hosting my parent’s mail as well and I want not that they get a warning box if they access my SMTPS or POPS servers. It is on http://www.halbheer.info/Transfer%20Documents/Technical%20Documents/halbheerroot.zip and is called Halbheer Root today.

This deployment allowed me then to go for Domain Isolation with IPSec and in parallel Network Access Protection. I will talk more about this, once I touched briefly on the theme of monitoring.

Roger

Before I start here: Let’s be clear that I will not say (and will never say) that if a customer was infected with Conficker he had a poorly managed network!

I had a lot of discussions over the course of time about the reasons for customers being infected. We all know the attack vectors of Conficker but what are the real reasons behind it?

• Poor or no Patch Management: We are coming back to my Russian Roulette post back in January. If you decide not to patch or leave it to the admin to decide, in my opinion this is negligent. And now, please, do not tell me that this is a Microsoft-only problem. Base don the Security Intelligence Report, we are responsible for 3% of the industry-wide vulnerabilities. So, do not forget to patch the other 97% as well!
• Unmanaged Machines: This comes down to compliance management as well and is not an easy problem to be solved. But we have seen customers who thought that they were fully patched just to find some unmanaged machines which were not – and these machines were the starting point of the infection. Let’s be clear on this one: This is a problem which can be addressed but it is a project you have to run with the corresponding investment. There is technology out there like 802.1x to keep them off the network or IPSec Auth to make sure they are not able to talk with your key machines – but you have to deploy them.
• Weak (or inexistent) passwords: Once you had (or have) Conficker on the network, weak passwords are often a pretty good vector for Conficker to spread. Again, it is about compliance management.
• Everybody is an Admin: We can now debate about this again, who is to blame with this. It is a fact that a lot of users run as Admins (yes, in Enterprises as well) because certain applications do not run without Admin privileges. The virtualization part of Windows Vista definitely helped to reduce this. Since Windows Vista (and Windows 7) I am not running as Admin anymore nor does anybody else in my network! This is once of the key achievements of UAC.
• Unsupported Operating Systems: It is really unbelievable how many NT4 we still find out there. We retired NT4 SP6a on 31.December 2004. Please do not come now and blame us for our policy. Today we support our products for 10 years at the supported service pack level (Business and Developer products). I know that there are reasons why you cannot upgrade – but there are a lot of machines out there which could be upgraded! Additionally it is sometimes worrisome to me how often I see old Operating Systems and unsupported applications being connected to the network without having any further protection and/or shielding.
• Anti-Malware Protection: This is a very difficult area now. There are customers who had Antimalware in place and did their best to having the signatures updated – however the AV-vendor failed to protect their customer base. We have been out with a signature for Conficker.B since December 29th – not only detecting Conficker.B but removing it. And there are some vendors talking very loud about Conficker did a very doubtable job.

I know that it is not easy but looking at the reasons above, I am convinced that a well-managed environment would have had a good chance to withstand Conficker. Well-managed meaning:

• Having proper policies in place where Business Risk Management is being seen as a fundamental part of an IT management
• These policies are enforced through administrative measures and audits as well as through technical means (yes, the auditor can be your friend).
• Violation of policy will be punished.
• Apply not “best practices” but good practices to your environment

It just showed us once more that running a network of a certain size is an engineering practice and not an art. Today’s economical situation does not help here either as a lot of companies want to save cost. However, a well-managed network to me is an inexpensive network as well – and a secure network! So, we should definitely think about this further. I am convinced that in today’s time we have to move from “best of breed” to “best of need” and in addition we have to make sure we deliver and you deploy a “best of need integrated platform” to address the challenges outlined above. So that you can concentrate on a business strategy as well as on processes!

This raises the 1 Million Dollar question: How the hell can you make sure that you know what you need. Well, you have to do Business Risk Management. A lot of companies – to me – miss the “business” in the statement above it they do Risk Management at all. From my point of view, it is not the CSO’s job to decide about the risks acceptable for a company. It is the Management Board. At the end of the day it is a business decision and not an IT decision! However, it is the CSO’s job to make sure the Management Board understands the risks they are taking on a level which is understandable for a business leader.

Let me add one final statement. Microsoft IT has a pretty tough job to do with all these geeks connected to the network running all sorts of beta software. However, I did not feel any disruption from Conficker. So, there is a good chance that they did an excellent job to keep it out. So, it is doable.

I will try to blog more about the platform in the near future. I started to bring these pieces together in my test environment to get some hands-on experience and I want to share more of this with you

Roger

I like that: As you probably know, I did a tour through the Gulf when we launched the Security Intelligence Report last year. One of the reasons was that we know that the Gulf has a pretty high malware infection rate. You can read this in the corresponding blog post: Security Intelligence Report v5 Live!

Now, QTEL (the ISP in Qatar) released an interesting document called Qtel’s Guide to a Faster Internet Experience. What I like about it is that most of it is about security but it actually addresses the user where it “really hurts”: Internet performance.

You can read it yourself at the link above

Roger