Windows Server Customer Engineering : Security

Scale testing the world’s largest PKI… all running on WS08R2 and Hyper-V

morello — Mon, 10 Aug 2009 23:50:00 GMT

This week, we've been in the Enterprise Engineering Center (EEC) doing our scale testing on a project to help build the world's largest PKI. When fully implemented over the next couple of years, this PKI will be the world's largest, issuing 100s of millions of certificates from 100s of CAs to devices around the world. The entire design is built on WS08R2 Hyper-V and WS08/WS08R2 CAs. Management is done using SCVMM.

In the EEC, we were able to simulate a portion of the hosting environment and drive load against it to find bottlenecks and optimize around them. To simulate one of the Hyper-V hosts, we used a similar machine to the ones being used the hosting facilities: a 2.4GHz, 4 socket, quad core machine with 64GB. We took a sysprep'd copy of the actual CA VM image used in the customer environment and loaded our host with 10 VMs, each assigned a single VCPU and 6GB. All 10 of these VMs were connected to an nCipher netHSM 2000. To generate load, each CA VM was paired with a single DC and 5 client machines, each assigned a single VCPU and 2GB and separated from the CA by a WAN simulator that added latency and throughput constraints based on the customer's actual network topology. We used an internal PKI test tool to have each client machine open 4 request sessions and requests 1000000 2K key certs per session.

After <24 hours, we'd issued >20 million certificates from this single physical chassis. During these tests, we found that:

Per VM CPU load was ~25%, total host CPU load was ~20%
Relatively little memory was required by the CA VMs, even at this high stress; thus we're optimizing the design to increase the density of CA VMs per chassis, to 30:1 (2GB per VM)
The performance bottleneck in this design is the HSM; as we increased the number of CA VMs being stressed, our requests per second per CA fell significantly, from >100 to ~18-20, giving a net issuance rate for the entire chassis of ~200 per second
When investigating the HSM, it became clear that it was the gating component (note that its 150 request queue on the left is persistently nearly saturated and CPU on the right is pegged at consistently at 85%)

Overall, this testing was a great validation of the performance of ADCS. Our software ran as fast as the HSM would allow and we gracefully handled response delays introduced by it. Also, the fact that we're able to run this configuration entirely on Hyper-V and get ~30 CAs per physical host provides an efficient scale story for even the very largest and most complex environments around.

Using the FREE network security tools you get in Windows (and who doesn't love free tools?)

pfetty — Fri, 20 Feb 2009 01:42:00 GMT

In my last post I talked extensively about the use of 802.1x for network authentication (wired or wireless) and talked about the benefits of the 2 common approaches to controlling machine access (VLAN vs. Port ACL). While 802.1x remains a very popular mechanism for controlling port based access for machines coming onto the network, it also has some significant requirements associated with it, primarily, having 802.1x capable switches and/or access points in place today. You would be surprised how many people think that their devices support this capability, but in reality, they may find that this is not the case. What this means is that if your hardware vendor tells you that what you have isn't 802.1x capable, be prepared for some sticker shock when you find out how much it will cost for an upgrade!

Given the state of IT budgets today, doing a massive hardware upgrade is probably not something you are prepared to do now, or any time in the near future. That being said, you still have the requirement of securing the endpoints that are on your network and ensuring that they are kept up to date with patches, AV signatures, Anti-Malware Definitions etc. These 2 efforts may seem to conflict, but they certainly don't have to.

Let me introduce you to Server and Domain Isolation (SDI) with IPsec, all built into Windows, all free of charge!

Now, as to not sound like a late night infomercial peddler of dodgy wares, I will explain what I mean by "free of charge". What this means is that if you have purchased Windows Server (any SKU or version) and a Windows Desktop OS (Windows 2000 or later) and an associated Client Access License (CAL) (you usually purchase a CAL when you buy a server from Microsoft), these technologies do not have any additional licensing requirements whatsoever.

Now that we have the cost definition out of the way, let's talk about what these technologies can do for you.

Server and Domain Isolation (SDI)

Simply put, SDI utilizes the IPsec (Internet Protocol Security) technologies that have existed in Windows ever since the days of Windows 2000. What SDI allows you to do is to create access policies based on Active Directory groupings that can require authentication between any set of machines in the network (or all machines). The tools for creating and distributing these policies are also built into Windows and policies are distributed via the Group Policy mechanism, meaning that yes, a machine must be joined to the domain to easily take advantage of SDI. Since we are talking machine and/or user authentication here, there are 2 options for credential use in SDI, they are X509 certificates, or a Kerberos ticket. There are advantages and disadvantages to the use of each of these credentials which I won't go into here, but the message is that there is flexibility here with both credentials being secure (i.e no passwords.)

Now, let's talk about one big option that 802.1x cannot offer you, and that is data encryption.

Many customers have never really ever considered doing any kind of encryption on their network because they probably never understood how easy it is to implement. Many think that there is some hardware requirement, or that their machines will come to a crawl performance wise, when in reality, neither are true. Granted, you have the option to create very stringent policies that require stronger encryption, but in many cases you simply won't need to do something like this, or may choose to bypass encryption all together.

Example

Here's a real world example of how SDI can be used:

The Microsoft network uses SDI technologies to protect our assets. Pat Fetty (aka me) is a known and trusted employee and is allowed to access the network from his office, or remotely every day. However, since Pat is not a Windows developer, pat is unable to access or even see or ping our source code servers. Pat is also unable to see things like our HR server, financial servers etc. How is that you ask, well, Pat is not in the proper security groups in AD to have the proper policy applied to him, or his machines, that allow this type of access.

Using this simple example, think of some of the other scenarios that you, or your customers may be faced with that you may be able to solve. Here are some others I have come across in my job in the WinCAT team:

- Allow only doctors to see patient records, and encrypt all traffic going to and from those servers

- Allow only attorneys to be able to see servers X, Y and Z

- Encrypt all traffic to and from an ATM machine to the banks home offices

- Encrypt all Active Directory synchronizations between servers where traffic is going over the Internet

- Authenticate all traffic from my Windows machines, but exempt traffic from my main frame machines and all Macintosh machines (we'll talk a bit more about this)

This is just a small sample list of real business problems that are out there that SDI can help to solve.

Now, getting back to the last item regarding mainframes (let's just say all *ix) and Macs. There are Ipsec stack implementations available for these platforms, but I have personally not seem them work. As previously mentioned, the Microsoft SDI set of technologies are based 100% on public RFC's so there is no reason to expect that any implementation that is RFC compliant wouldn't work with any Windows machine.

Another reason why this is important is because the tools to build and manage SDI policies allow you to create very simply policies to address this scenario. For instance, you can have a policy that looks like this:

If you are a machine in group A, and you are capable of negotiating IPsec, then negotiate IPsec with a (Kerb ticket or cert)

else

Allow traffic to flow in the clear

You may be asking yourself "what kind of security policy is that" and the answer is that it is one that doesn't disrupt the flow of traffic on the network, and yet will negotiate the security you dictate when possible. This is a very important concept since the reality is that your machines today are most likely not doing any kind of SDI today, so by having your policies based on AD group membership, and by having a "best effort" type of policy like this you can ease this type of technology into your network with little or no disruption!

SDI versus 802.1x

We talked about this at the beginning and I think that SDI is the hands down winner here given that if you are a Windows environment

Deployment

To do 802.1x means you will have to touch all the switches and AP's in your network and configure them to do 1x on their ports, and to then send all traffic to a back end authentication server. In many cases, the group that manages desktops and servers in your company is not the same group that manages the infrastructure, so you now are getting more people involved in the effort.

SDI utilizes Group Policy and other built in tools to distribute policies and credentials, and the best part about this whole thing, is that SDI will work across whatever hardware you have in place today!

Management

This is a bit of a toss up, but I'd give SDI an edge since you can use a variety of methods to manage who is on the network accessing resources. 802.1x usually requires an enterprise ready SNMP (Simple Network Management Protocol) solution either from our hardware vendor or from a third party

Security

This is the one that gets the networking guys feathers ruffled, but I give SDI the advantage here for a couple of reasons. First, SDI offers you the ability to encrypt traffic which is something that 1x cannot offer. Second, in 802.1x, once you get passed the switch you are on the network doing what you please until you either reboot, or the switch kicks you off the port. SDI allows you to control the credential that is given to that client so that if for some reason you need to remove the client from the network, you can more quickly do so via SDI than 1x.

Overall I feel that both technologies here offer up some nice benefits (we didn't even touch on Guest Access which I feel is better done with 802.1x than SDI), but in the end, you should evaluate what your business needs are and make your technology decision on that. Usually the second phase of that decision process is cost, and that is where using the tools we have built into Windows can save you loads of money down the road assuming you plan on running Windows for the foreseeable future (which we hope that you are of course!)

With budgets shrinking and/or disappearing I would encourage everyone to take a look at what you have purchased in the Windows platform and ask Microsoft how to best utilize it because we are definitely here to help you get the most out of Windows!

Deployment Security Designs for Forefront IAG/UAG Virtual Appliances

morello — Wed, 13 Aug 2008 19:34:00 GMT

One of the most compelling capabilities being added in IAG SP2 (which will also be available in UAG) is the 'virtual appliance' installation option. A virtual appliance is a preconfigured, ready to use Virtual Machine that already has Windows Server and IAG / UAG installed. Microsoft will build the VHD and make it available for customers to download. Customers will then take the Virtual Hard Drive (VHD) and drop it into a child partition on a Hyper-V host. At this point, the VM would function like a classic IAG installation, with all the normal features and capabilities customers have come to expect. The reason we've added this capability in IAG is to give customers options for how they want to deploy IAG in their networks. For many customers, the pre-tuned, dedicated hardware appliances available from our partners are a great option that fit in well with their overall management methodology. For other customers, they prefer a more standardized hardware platform in their datacenters and thus the virtual appliance on Hyper-V is preferred. Note that it's not a question of which is 'better'; the two options allow customers to chose the solution that best fits their environment.

For customers looking at deploying the virtual appliance, a common question is what is the best way to provide a secure virtualization environment for the IAG/UAG VM? There are three primary design options to choose from. Again, it's not a question of what option is best; rather, customers should look at each model and decide which best aligns with their management approach.

Option 1: Classic Physical Appliance

It may seem strange to list a physical appliance as an option here, but arguably the dedicated physical appliance is the most hardened configuration out of the box. The reason for this is that the OEM appliance vendors take Windows Server and IAG and really mold the entire hardware platform around them. In doing so, they reduce the attack surface of the machine by disabling services not critical to IAG, ensure necessary updates are installed, and then put that image on top of a hardware platform designed for them. Because IAG is built on top of Windows Server, it's possible for a customer to take many of the same software steps the OEMs do, but the benefit of the appliance is that it's all been done and tested for you. For customers looking for the most secure out of the box experience with IAG, physical appliances provide some unique benefits.

Pros: minimal configuration; pre-hardened operating system; hardware designed specifically for remote access gateway
Cons: limited hardware choice; potentially non-standard device and software configuration in an otherwise rationalized datacenter

Option 2: VM on Dedicated Hardware

While one of the key benefits of virtualization is the ability to run multiple operating systems simultaneously on the same physical hardware, it's by no means a requirement that a Hyper-V machine have more than 1 child partition. In other words, it's fully supported to run a Hyper-V system with only a single child. Why would you do this? If you want to have the manageability benefits of virtualization, but have workloads that can scale up and maximize an entire physical server, this approach is an effective model for getting the best of both worlds. Particularly when you use the Server Core option of Windows Server 2008 to run the parent partition, you have very minimal overhead incurred by doing so. In fact, key Microsoft web sites like TechNet and MSDN use this exact model in their production environments. When you think about this model for hosting IAG, the benefits are that you don't have concerns about resource contention between VMs (though Hyper-V has resource management controls available) and you don't have to worry about sharing the remote access gateway physical platform with any other workloads. Because Hyper-V supports the same huge catalog of server hardware that Windows Server 2008 does, you have great flexibility in what the physical layer looks like. Whether you prefer 1U, 2U, blades, and regardless of OEM, you'll be able to easily integrate the Hyper-V host and its IAG child partition into your existing datacenter. Finally, because you can use whatever hardware you prefer, it's easy to place the server wherever it needs to go within your network. For example, it is often easier to provision a new blade into the DMZ network to host IAG than it is to securely route traffic from the DMZ to a larger virtualization system in the internal network.

Pros: great choice in hardware; can use existing organization standards for hardware and operating system images; with Server Core, very low overhead for parent partition; great flexibility in network placement
Cons: may require greater setup effort to configure hardware and parent partition operating system

Option 3: VM on Existing Virtualization Environment

For customers that already have a Hyper-V environment, they may wish to simply add the IAG VM to the existing hosts. This is particularly true if a customer has already invested in building a highly reliable, well tuned hosting environment, using tools like Failover Clustering. In these cases, there's no problem with running IAG in a child partition on an existing physical server already running other VMs. So long as the traffic is properly routed to the VM, IAG can function perfectly well in such a configuration. However, when sharing physical resources with other child partitions, it's particularly important to allocate sufficient capability to the IAG VM. This should be done both by allocating enough memory and CPU capability to VM, as well as ensuring that Hyper-V prioritizes requests through the IAG VM appropriately. Additionally, there are significant performance and security benefits to dedicating physical network adapters solely to the IAG VM, rather than sharing them with other VMs. Having dedicated NICs ensures that IAG will not need to compete for network IO and simplifies the routing of remote access traffic to and from the VM.

Pros: efficiency of reusing existing investments in Hyper-V physical platform, such as Failover Clustering
Cons: more planning required to ensure sufficient resources for IAG child partition; potentially more complex network routing needs if the existing environment does not already receive traffic from internet hosts

Virtual appliances are all about customer choice; providing you with the right options for security and placement while allowing you to chose your own hardware platform or reuse one you already have. There's no right choice that applies to all situations, so think about your environment and goals, and chose the option that fits your network best.

The Definitive Guide to NAP Logging

morello — Mon, 29 Oct 2007 15:39:00 GMT

Pete Rivera is the Windows Team Lead on one of our DoD support teams and we've been working together on a NAP project. In addition to being a master of style and male fashion, Pete also puts together some great guidance for his customers. Recently, he wrote a detailed description of all the various logging capabilities that you might ever need to use to debug a NAP problem. Thanks, Pete!

NPS has various places where it does logging and/or creates a log… First off we do accounting IAS logging of the NPS status and network connection process data in %windir%\system32\LogFiles, but it can be configured to an alternative location. The log is:

IN<date>.log

2. Secondly we also can do SQL logging to a SQL 2k or SQL 2k5 database. This is used for logging user authentication and accounting requests: Logs user authentication and accounting requests in a stored procedure in a SQL Server 2000 or SQL Server2005 database. Request logging is used primarily for connection analysis and billing purposes. It is also useful as a security investigation tool, providing a method of tracking down the activity an attacker.

3. Likewise you can enable debug trace logging via netsh and this can be used to help provide detailed information about the Network Policy Server operation when NAP policies are configured: Netsh ras set tr * en

%windir%\Tracing\IASNAP.log

4. In addition this enabled a slew of other IAS/RAS related logs in the same folder (i.e.: IASSAM.LOG, IPSEC etc ):

%windir%\Tracing\*.log

5. You also have Event Logs. These provide a lot of info about the operation of NAP and connecting clients but is used primarily for auditing and troubleshooting connection attempts. Depending upon your build they are either in the SYSTEM (B3) log and/or the security log (RC0). There is also the Network Access Protection event log which you'd find on NAP clients.

6. On the client side we can enable NAP client Debug Tracing logs as well. This is enabled either via netsh or via the NAP client Configuration snap-in. It's an ETL file which is generated only by using logman… so you'll need to do a logman start QAgentRt -p {b0278a28-76f1-4e15-b1df-14b209a12613} 0xFFFFFFFF 9 -o %systemroot%\tracing\nap\QAgentRt.etl –ets in order to turn start .etl generation.

7. likewise we can also do WHSA tracing for NAP also… the trace GUID is 789e8f15-0cbf-4402-b0ed-0e22f90fdc8d

8. DHCP QEC tracing…

Netsh dhcpclient trace enable. This command enabled QEC tracing and the trace files will be generated at %WINDIR%\System32\LogFiles\WMI\DHCP*.*

9. EAPHost Tracing for 802.1x

Trace logs containing debugging information can assist users in finding the root causes of issues that occur during the EAP authentication process. The debugging information can include API calls performed, internal function calls performed, and state transitions performed. Tracing can be enabled on both the client side and the authenticator side.

When EAPHost tracing is enabled, logging information is stored in an .etl file in a user-specified location. Tracing generates an .etl file.

10. EAPHost Tracing for 802.1x (client side)

To enable tracing on the client side:

Run the following command: logman start trace EapHostPeer -o .\EapHostPeer.etl -p {5F31090B-D990-4e91-B16D-46121D0255AA} 0x4000ffff 0 -ets

Run the following command: logman stop EapHostPeer -ets

Convert the etl file into text using the following command: tracerptEapHostPeer.etl –pdb <pdbpath> -tp <tracemessagefilesdirectorypath> -o EapHostPeer.txt

11. EAPHost Tracing for 802.1x (Authenticator side)

To enable tracing on the authenticator side:

Run the following command: logman start trace EapHostAuthr -o .\EapHostAuthr.etl -p {F6578502-DF4E-4a67-9661-E3A2F05D1D9B} 0x4000ffff 0 -ets

Run the following command: logman stop EapHostAuthr -ets

Convert the etl file into text using the following command: tracerptEapHostAuthr.etl –pdb <pdbpath> -tp <tracemessagefilesdirectorypath> -o EapHostAuthr.txt

12. The we have the SCCM related logging specific to the SCCM SHA and shv. The Configuration Manager 2007 client computer log files are found, by default, in %windir%\CCM\Logs. For client computers that are also management points, the log files are found in %ProgramFiles%\SMS_CCM\Logs.

13. Ccmcca.log

This file logs the processing of compliance evaluation based on Configuration Manager NAP policy processing. It also contains the processing of remediation for each software update required for compliance.

14. locationservices.log

This log is used by other Configuration Manager features (for example, information about the client's assigned site), but it also contains information specific to Network Access Protection when the client is in remediation. It records the required remediation servers (management point, software update point, and distribution points that host content required for compliance), which are also sent in the client statement of health.

15. SMSSha.log

This is the main log file for the Configuration Manager Network Access Protection client, and it contains a merged statement of health information from the two Configuration Manager components: location services (LS) and the configuration compliance agent (CCA).

This log file also contains information about the interactions between the Configuration Manager System Health Agent and the operating system NAP agent, and also between the Configuration Manager System Health Agent and both the computer compliance agent and location services. It provides information about whether the NAP agent successfully initialized, the statement of health data, and the statement of health response.

16. CIAgent.log

This tracks the process of remediation and compliance. However, the software updates log file, Updateshandler.log provides more informative details on installing the software updates required for compliance.

17. SDMAgent.log

This log file is shared with the Configuration Manager feature desired configuration management, and it also contains the tracking process of remediation and compliance. However, the software updates log file, Updateshandler.log provides more informative details about installing the software updates required for compliance.

On the server side for the System Health Validator point, you should first check the Windows Application event log on the Windows Network Policy Server computer. This log will record any failure categories and errors with the source being SMS_SYSTEM_HEALTH_VALIDATOR. These are also raised as Configuration Manager status messages. Otherwise More detailed logging information can be found in the Configuration Manager logs and the System Health Validator point log files are located in %systemdrive%\SMSSHV\SMS_SHV\Logs.

19. Ccmperf.log

This log contains information about the initialization of the System Health Validator point performance counters.

20. SmsSHV.log

This is the main log file for the System Health Validator point. It logs the basic operations of the System Health Validator service, such as the initialization progress.

21. SmsSHVADCacheClient.log

This log file contains information about retrieving Configuration Manager health state references from Active Directory Domain Services.

22. SmsSHVCacheStore.log

This log file contains information about the cache store used to hold the Configuration Manager NAP health state references retrieved from Active Directory Domain Services, such as reading from the store and purging entries from the local cache store file.

23. SmsSHVRegistrySettings.log

This log is used to record any dynamic changes to the System Health Validator component configuration while the service is running.

24. SmsSHVQuarValidator.log

This log file records client statement of health information and processing operations. To obtain full information, change the registry key LogLevel from 1 to 0 in the following location:

HKLM\SOFTWARE\Microsoft\SMSSHV\Logging\@GLOBAL

25. <InstallationPath>\Logs\SMSSHVSetup.log

This log file records the success or failure (with failure reason) of installing the System Health Validator point.

NAP Customer Web Cast

morello — Thu, 28 Jun 2007 01:10:58 GMT

Jeff Sigman (NAP Release Manager) just added a post I wrote to the NAP blog about an upcoming customer webcast. If you weren't able to make it to TechEd, you probably missed the session that fellow WinCAT PM Pat Fetty did with Hunter Ely, who is charge of Louisiana State University's IT security technology and is one of our NAP TAP customers. Pat and Hunter's session covered LSU's architecture, deployment process, and lessons learned. So, if you've wanted to hear some real world experiences on NAP, tune into the web cast.

Morello

WinCAT the recording artist

morello — Fri, 11 May 2007 18:39:00 GMT

I've written a number of articles for TechNet magazine over the past year (http://search.live.com/results.aspx?q=morello+site%3Amicrosoft.com%2Ftechnet&mkt=en-us&FORM=LVCP) and they recently interviewed me for their monthly podcast series. The article from the April issue was focused on the Security Configuration Wizard and the first part of the podcast covers that. We also talk about smart cards, HSPD-12, and how we used Groove during Katrina. Check out the podcast here: http://www.microsoft.com/technet/technetmag/podcast/default.aspx. My part starts around the 9 minute mark.

-John Morello, WinCAT security PM

WinCAT on Channel9

morello — Fri, 20 Apr 2007 05:23:50 GMT

I'm John Morello, the WinCAT PM for security and anywhere access which encompasses a lot of interesting Microsoft technologies, like Terminal Server, Forefront Edge, and the Certificate Server. I've been a frequent contributor to TechNet magazine on these topics so you may have read some my articles. Just a couple of hours ago, I was interviewed by Charles Torre at Channel9 to discuss PKI and crypto improvements in Vista and Longhorn. There's a long list of improvements, including built in OCSP support, CA clustering, CAPI2 diagnostics, and SCOM management pack support, just to name a few. I'm working on an article for a future issue of TechNet to cover these topics in more detail and we'll update the blog once it's online. I'll also post once Charles puts the video online.