The common criteria IT Security evaluation process is a powerful, useful, well thought out, and well organised system from a technical perspective. To a deep techie it means that there should be some level of assurance that a particular product can stand up to a serious degree of scrutiny, was developed relatively soundly, and has good secondary characteristics like access to its source code and design methodology. No one should ever try to detract from that. From my perspective, I was one of the annoying "field" guys that pushed our ISA Server product team to do this certification, and feel very glad that they did. It is a significant investment for a company to undertake a Common Criteria evaluation (just ask anyone who has ever gone through it) and at Microsoft, we certainly took this project seriously, and both ISA Server 2000(EAL2), and ISA 2004 (EAL 4+), as well as Windows 2000, and Windows 2003, and XP all have CC certifications, as does Exchange 2003 with SP1 (EAL 4+). Microsoft is proud of these certifications, and has devoted significant resources for us to work with these industry bodies to attain independent certifications.

The sceptics will point to the fact that Microsoft can get CC as proof the programme is bad, apparently they view this as an exclusive club that the "insecure" shouldn’t join - a good example of this is here, in Bruce Schneiers blog, he just asks the question, the comments in it speak for themselves - it is a great discussion to read.

So what’s wrong with this whole thing? Well like anything in the security business CC is often (and usually) used to establish that the product is "secure", and that using it will make you secure. The good salesman will point out regulatory or governmental regulations sometimes obliging the use of a CC programme device, and it is engrained in the consciousness of the security world that a CC sticker means buy the product, which was never the intention of the CC programme in the first place. The techies among us will instantly point out that security is about the thinking and quality of the deployment, and its architecture, not just the collection of stickers on its constituent parts. For us to understand why salespeople get away with it, we need to understand roughly how CC works.

So here goes - when you submit a common criteria evaluation you have to decide what level of assurance you want your product to be evaluated to, according to UK CESG the following definitions apply from EAL1 to EAL7(highest)- click here  . For commercial software, EAL 4 is the highest practical level of assurance, you can get EAL 7 as a commercial product - check out here for proof, but part of the EAL 5-7 evaluations means that commonly used and known algorithms or those that don’t pass stringent criteria can’t be used (like SSL, or other ones the "normal" world today cant live without). In addition the higher certifications also require a high degree of simplicity in the design ruling out most commercial software which is classified as "complex". So beware myth number one - sales guy says "my product is CC so sign here on the order form". You need to know what level of assurance was tested. (and a few more things too, read on)

The next one gets also wildly confused, and that is the concept of a "protection profile", and the concept of a "Security Target". A protection profile is a "group" in windows terms of features that does one logical thing, for example a "strict firewall" which contains several features or characteristics that constitute that definition, like stateful inspection, dropping unsolicited packets etc. The Common Criteria site has a list of their profiles, and what feaures are in each – access it here, my favourite is “waste bin management” J . A lot of vendors find it easy to take one or two protection profiles and submit their product for evaluation. So CheckCoat takes their CisScreen 10000 appliance and submits it for assurance level 4+ for “strict firewall” – which means the EAL4+ level and depth of analysis will be used to evaluate the 12 strict firewall features of the CisScreen 10000. ONLY THE FEATURES IN THE PROFILE (GROUP OF FEATURES) would then gain CC if passed. NO OTHER FEATURE, if you enabled a bunch of other things, some of which may be on by default in the product, then most likely the certification would also be invalidated, unless only the features evaluated in that group were used.

When I go into customers I find that around 80% of the Common Criteria stickers on the outside are usually invalidated by the configurations or deployment of the devices, usually for a good reason, because the device was next to useless in its evaluated form, if you see this in a device you are buying, it’s a strong signal the company is doing CC to keep up with the sales angle. Most CC discussions at a product company have to involve a cost/benefit analysis of how much revenue you would expect to get, versus the cost (significant) in time, money, and human capital to undergo the certification, cutting the security target short is a way to lessen this pain.

So beware and figure out what features are in your evaluated Security Target and what ones aren’t. Usually the boss just says CC is required, so you only look at stuff with CC – not realising its not a simple box to tick. This brings us to the concept of Security Target, which is a list of all the features that were evaluated for (whether or not they are part of a grouping of protection profiles). Simply put a Security Target is every single feature that the vendor included in the evaluation, and thus would be certified in the evaluation. Many new features aren’t part of an existing Protection profile, so the vendor says my new cool feature (let’s call it Outlook Web Access Protection) isn’t in any firewall protection profile, so I will include the feature anyway on its own. So if the sales guy says “ISA didn’t pass the firewall protection profile so its not a firewall” doesn’t mean to say it didn’t evaluate as a firewall, just that Microsoft chose to put in custom features (usually more than in the simple profile) that didn’t exist in a grouping. Like we did when we added Outlook Web Access Filtration, and our RPC filter into our Security Target, so if you use ISA for RPC protection its still CC, if you use almost any other firewall’s RPC filters (the few that have one) you would lose your CC. For a good document that we wrote on what is in ISA’s evaluation criteria you can check here . So before you buy a common criteria product you should do two things –

1 – Check how you will deploy the product – and ensure that the entire infrastructure is going to be secure, don’t assume that a CC sticker will mean instant security.

2 – Check here for the list of what level the product has been reviewed at, and WHAT FEATURES (security target) were reviewed. Open the documents for your product and read the public sheet of what the sec target was, and more importantly, what wasn’t in the evaluation.

Check out www.commoncriteriaportal.org as a great place to learn more

What do you think - leave your comments - Should Microsoft continue pursuing CC ? Do you think it adds value (I do)?, What other certifications should we strive for ?