March, 2009

Ned here again. Despite the title, today’s post is not about Australopithecus. I recently spent a few days in Redmond teaching part of the Microsoft Certified Master course. During the class… eh wait…

You’ve never heard of MCM? You should definitely check it out if you’re looking to take your career to the next level, and if you like really, really hard final exams.

Ok, back on track. During the class breaks one of the students asked an interesting question: “Why does the DFS client not return a complete list of DFS links in a referral request?”

If you’ve never looked at a very large DFS Namespace link referral list before, you might not have noticed this either. For most companies, there’s only one really big list – SYSVOL. As you know, the Domain System Volume (SYSVOL) share is actually a special-cased DFS link that only runs on domain controllers and doesn’t require any special set up other than DCPROMO to configure. So let’s consider the following:

Here I run DSQUERY on a Vista SP1 client with RSAT installed to list out all the DC’s in the domain:

For those that don’t feel like counting, that’s 45 DC’s in this domain. So then I run the DFSUTIL command to list out the link targets in this SYSVOL link:

So I have 45 DC’s, but I only see 29 DC link targets here. That’s weird. Maybe it’s just DFSUTIL. Let’s get a third opinion from a network trace like a good engineer always does:

Here’s the referral request packet from the client:

And here’s the response packet from the DC (i.e. the DFS root namespace server):

Note how the DC has returned 29 servers as well, so there was nothing wrong with DFSUTIL. The network trace shows how the referral indexes match up perfectly with what DFSUTIL shows, and the Index:1 value matches the position 0 in the DFS referral list we saw earlier. 29 is too weird of a number to be hard coded; developers like binary multiples. So what gives?

After source code review and a confirmation chat with the (awesome) lead developer for DFSN, I had my answer: This is expected, by design behavior. A Windows client will only request a 4 kilobyte buffer of links from the Namespace Root server. The 29 link list is not a hard-coded limit, but just happens to be how many links here in this domain will fit in a 4K referral request buffer used by the client. If the link paths were shorter then more would fit (and mine are pretty long here compared to most domains).

This buffer is used for performance reasons - not memory performance, but failover time performance. Since DFS lists are serialized and each server must be tried if the preceding server is unavailable, it was decided many years ago that only 4K would be allocated. Assuming a failover time of ~20 seconds (including retries) per link target, having an extremely long list of links would make the user experience very poor if the client had to fall through 500 servers and fail just because there was a temporary network issue on the client. If this many links are unavailable, it's highly likely that the network for this client is having catastrophic issues and even if a complete referral list was provided it is unlikely the link target requests would be fulfilled. And since until now this was never even documented, it seems like the decision worked out fine. :-)

For the curious:

1) No, this 4K buffer is not configurable.

2) The SPC referral buffer is 56K, so it's unlikely that it will ever be filled in all but the most gigantic forests. We’ve never heard of it happening in 10 years of AD, at least.

And to that student and his excellent question – Viele Grüße!

- Ned ‘N+1’ Pyle

Oxymoron - a figure of speech by which a locution produces an incongruous, seemingly self-contradictory effect, as in “cruel kindness” or “succeeded with errors.”

Hi, Ken here. Recently I encountered an issue where the customer was trying to install certificate services on Windows Server 2008. They installed the Active Directory Certificate Services role using Server Manager. When the install completed they received the error “Installation succeeded with errors.” Sometimes “completed with errors” isn’t a big deal, but in this case it was really just “failed.”

So first off, let’s take a look in the Event Log to see if anything more informative than “Succeeded with Errors” has been recorded. In the event log I saw the following events:

Log Name: Application
Source: Microsoft-Windows-CertificationAuthority
Date: 6/25/2008 11:06:09 AM
Event ID: 5
Task Category: None
Level: Error
Keywords: Classic
User: SYSTEM
Computer: con-root-ca.contoso.com

Description:
Active Directory Certificate Services could not find required registry information.
The Active Directory Certificate Services may need to be reinstalled.
Log Name: System
Source: Service Control Manager
Date: 6/25/2008 11:50:57 AM
Event ID: 7023
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: con-root-ca.contoso.com
Description:
The Active Directory Certificate Services service terminated with the following
error:
Cannot complete this function.
Log Name: System
Source: Microsoft-Windows-DistributedCOM
Date: 6/25/2008 11:41:13 AM
Event ID: 10016
Task Category: None
Level: Error
Keywords: Classic
User: CONTOSO\testuser
Computer: con-root-ca.contoso.com
Description:
The application-specific permission settings do not grant Local Launch permission
for the COM Server application with CLSID
{D99E6E73-FC88-11D0-B498-00A0C90312F3}
to the user DOMAIN\username SID (SID Value) from address LocalHost (Using LRPC).
This security permission can be modified using the Component Services
administrative tool.

So, from the event logs, we see some pretty serious issues. Certificate Services cannot find required registry information, and is causing the service to not start. When we try to start the service manually, we get the following error:

Cannot complete this function 0x3eb (win32:1003)

That’s a nice error, but it doesn’t tell us anything new. The other event referenced the registry, so, let’s go take a look in there. The keys for the Certificate Authority are located in the following location:

HKLM\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration\<CA Name>

In the registry we see that the “Setupstatus” value is set to “0xc-“. This is not correct, if the installation was really successful, it should be set to “1”. We also see that the “Security” value is missing completely. The security value is a default binary value that is stamped in the registry during the Certificate Authority installation.

Setupstatus = 0xc –
Security=?? – This value was missing completely

So now we know that the installation not only had errors, but actually did not complete. So, I then moved over to the Server Manager log, to see what it had to say. The Server Manager log contains details about operations performed by the Server Manager. The log can be found in %SystemRoot%\logs\ServerManager.log.

In the ServerManager.log, I saw the following:

2124: 2008-07-01 14:05:50.494 [Provider] Error (Id=0) System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) at Microsoft.CertificateServices.Setup.Interop.CCertSrvSetupClass.Install()
at Microsoft.Windows.ServerManager.CertificateServer.CertificateServerRoleProvider.Configure(Object clientContext, String hostComputerName, XDocument guest, String guestIdentity)

So, in the Servermanager.log, we see a great big “Access is Denied”. That’s something I can understand, but it doesn’t tell us what it is trying to access. Well, we were installing Certificate Services, so I then moved over to the Certificate Services setup log. The setup log is helpful for troubleshooting in the event of errors. The setup log is located in %Systemroot%\CertOCM.log.

In the CertOCM.log, I saw the following:

109.2552.443: Install Server: Access is denied. 0x80070005 (WIN32: 5)0.0.965: Message Box: EmergencyMessageBox: Fatal: 0x80004003 MsgId: 0x : Access is denied. 0x80070005 (WIN32: 5)
114.5848.949: End: CCertSrvSetup::Install: Access is denied. 0x80070005 (WIN32: 5)

So in the CertOCM.log, I see another “Access is Denied”. This is still not specific enough to determine the issue. I know that during the installation, we look in the Active Directory to load the templates. So I then moved over to the CertEnroll.log to see what we have in there. When an enrollment attempt is made, the information is recorded in the %Systemroot%\CertEnroll.log.

In the CertEnroll.log, I saw the following error for all of the templates. Here is one of the entries I see, this one for the SmartcardLogon template:

808.4098.0: 0x32 (WIN32: 50)
808.4131.0: 0x32 (WIN32: 50)
429.2157.0: 0x32 (WIN32: 50): 00002098: SecErr: DSID-03150E8A, problem 4003
(INSUFF_ACCESS_RIGHTS), data 0
808.4138.0: 0x32 (WIN32: 50)
808.7585.0: 0x80072098 (WIN32: 8344): SmartcardLogon

I also saw the following error in the Certenroll.log:

202.5252.0: 0x80090016 (-214689380): Exception at
d:\rtm\ds\security\services\ca\fs\crypto\capicryptofactory.cpp(501):
CryptAcquireContext( &hProv, pszContainer, pszProvider, nProviderType, fAcquire)
HRESULT = 0x80090016

So this is something good. I was already sure that there is a permissions issue, but now we are seeing specifically, “INSUFF_ACCESS_RIGHTS”, when accessing the Certificate Templates in the Active Directory. So I then moved over to the AD, to see what kind of permissions we had on the templates. I did the following to check the permissions:

1. Using Adsiedit, I looked at the permissions on the templates under cn=public key services,cn=services,cn=configuration,dc=< Forest Root Domain>. The Enterprise Administrators group should have had Full Control on all of the certificate templates.

2. I also checked permissions for all the sub containers underneath Public Key Services. The account installing the CA should have had permissions. For example, Certificate Templates, OID, KRA containers.

Note: By default only the Enterprise Admins and the Domain Admins groups in the Forest Root Domain have enough permission to install and configure an Enterprise Certification Authority. If you have modified permissions, you may not have given yourself enough permission to do the install.

In this case, the permissions for Enterprise Administrators were set to Read-Only, or had been removed completely on some of the objects. Once we restored the Enterprise Admins back to the correct permissions, we were able to successfully install the Certificate Authority using the Microsoft CSP…and all was right in the World. :-)

Finally, here are a couple of links to some great information. The first is a Step by Step, complete with requirements and a lab setup to play with. The second is the Active Directory Certificate Services Portal site, loads of PKI goodness.

Windows Server Active Directory Certificate Services Step-by-Step Guide http://technet.microsoft.com/en-us/library/cc772393.aspx
Active Directory Certificate Services
http://technet.microsoft.com/en-us/library/cc534992.aspx

- Ken “The Beard” McMahan

Hey ladies and gents, Sean here again. Recently I ran into an issue with Windows Server 2003 that caused the Certificate Template Manager to hang. I’ll discuss the problem and provide solutions so you don’t get stuck wondering what’s going on if this happens to you.

First, let’s talk about the symptoms. If you try to open the Certificate Template Manager (certtmpl.msc) in the affected forest, it will not open…on any computer in that forest. If you look at task manager after attempting to open certtmpl.msc you will see mmc.exe consuming 100% of the CPU (50% if you’ve got 2, 25% if you have 4, etc.) regardless of how long you let it run. The only way to stop it is to end the process.

So, what’s going on when this happens? Most likely you have over 1000 Object Identifiers (OIDs) in the container CN=OID,CN=Public Key Services,CN=Services,CN=Configuration,DC=your,DC=domain.

When you open the Certificate Template Manager, it uses LDAP to enumerate all of the objects in this container. If there are over a 1000 OIDs then the default MaxPageSize is exceeded and the Certificate Template Manager will hang. The reason this happens is because a paged LDAP query is not used when looking up the OIDs in the OID container.

So, how do you end up with over a 1000 OIDs in the OID container? That’s a great question. To help answer that question I’ve listed a number of facts concerning the creation and deletion of certain objects.

1. Every template, issuance policy, or application policy (Enhanced Key Usage) that you create will create an OID in the OID container.

2. If you delete a template that you created, the corresponding OID will NOT be deleted from the OID container.

3. There is no way to delete issuance or application policies using the Certificate Template Manager. In order to remove either object you must delete the OID associated with it (determining which OID is associated with which policy is discussed later). Once the OID is deleted the Certificate Template Manager must be closed and reopened before the policy will no longer appear.

4. As soon as you right click on a template and select Duplicate Template in the Certificate Template Manager an OID is created in the OID container. If you decide you do not need to duplicate the template and hit Cancel instead of OK then the OID will remain but the DisplayName attribute on the OID will not be set (this is important when identifying which OIDs and templates are associated with each other).

Now that we know how OIDs are handled in certain situations, it isn’t hard to imagine what might cause you to have over 1000 OIDs in your OIDs container. It is possible that you just have that many objects, but it’s more likely that OIDs have not been cleaned up after templates have been deleted. It’s also possible that someone accidently duplicates the wrong template and hits cancel so it isn’t created.

Now that we know what’s causing the issue, how do you go about fixing it? There are two ways you can alleviate the issue. If you know you have a lot of certificate templates, issuance policies, or application policies that are no longer needed, or if you’ve recently added a lot of templates after deleting unnecessary ones from the Certificate Template manager, clean up the OIDs associated with the deleted or unnecessary objects. DO NOT RANDOMLY DELETE THEM! Find out which OIDs correspond to which objects and make sure you do not need it before deleting it.

To help determine which OIDS and objects are associated with each other, take a look at the output of this dsquery command:

dsquery * “CN=OID,CN=Public Key Services,CN=Services,CN=Configuration,DC=domainname” –scope subtree –filter “(flags=1)” –attr cn msPKI-Cert-Template-OID displayname >dsqueryfortemplates.txt

This command can be used to output the OID, the CN, and the DisplayName attribute of all of the templates in the OID container. Because we’re specifically filtering for objects with the flags attribute equal to 1 only OIDs pertaining to certificate templates will be returned. The output will look similar to Table 1.

Table 1

Now that we have this output, we know that OID 1.3.6.1.4.1.311.21.8.6650940.8038262.8601989.12750652.7494247.201.7982734.6580384 belongs to the Test User template. Also, notice that the OID ends with the same number that the CN starts with (6580384). If you decide this is a template that is no longer needed then the CN can be deleted in ADSIEdit for example.

*Remember, if there is no DisplayName attribute it’s likely the scenario described in number 4 from above has occurred.

Additionally, to help identify Issuance policies and Application policies, we can search for OIDs that have the flags attribute equal to 2 or 3 respectively.

dsquery *  “CN=OID,CN=Public Key Services,CN=Services,CN=Configuration,DC=domainname” –scope subtree –filter “(flags=2)” –attr cn msPKI-Cert-Template-OID displayname

This command will output the same information as the previous command (but for issuance policies, not templates) so you can make a decision as to whether or not the issuance policy OIDs should be deleted.

For your convenience I’ve included a table that lists the end of the OIDs that exist by default on a newly installed Windows Server 2003 Enterprise CA. The first 7 are templates and the last 3 are issuance policies. There are no application policy OIDs in the OID container by default.

Table 2

Once you’ve gotten the number of OIDs in the OID container below 1000, you will be able to open the Certificate Template Manager. Realize some cleanup will need to be done for the OIDs that were deleted. For example, if an OID for an Issuance policy called Test Issuance Policy was deleted in Active Directory and that issuance policy was applied on a template called Test EFS Recovery Agent, then when you look at the properties on Test EFS Recovery Agent you will see an OID in place of the display name for the issuance policy. Notice Figure 2 shows the Test Issuance Policy is applied. Once you delete the OID from Active Directory the display name will no longer show up in the GUI. Instead you’ll just see the original OID. This can be safely deleted if you’ve deleted the OID in Active Directory.

If you decide that all of the OIDs are needed in your environment then there is another, less desirable option. In order to get the Certificate Template Manager to open you will need to increase the MaxPageSize value to something LARGER than the number of OIDs you have in the OIDs container. Please realize that this will affect all of the DCs in your environment. If this value is increased too high it could cause performance issues. For more information, check out

How to View and set LDAP Policy in Active Directory by using Ntdsutil.exe -
http://support.microsoft.com/kb/315071

Now that you know how OIDs are handled in your Enterprise PKI environment you are better equipped to avoid problems that may arise from having a large number of OIDs in the OID container. Just remember to make SURE that you know what object an OID corresponds to BEFORE deleting it. It’s also a good idea to make sure that the object is no longer needed :-).

- Sean “Lurch” Ivey

Ned here. Our colleagues in Japan have started a new platforms support blog. Make sure you give them a read if you're supporting Windows anywhere in Nippon. :)

http://blogs.technet.com/jpntsblog/ 

- Ned Pyle

New KB articles related to Directory Services for the week of 2/21-2/28.

Ned here again. We're all snowed in down in Charlotte today, but that doesn't stop the blogging. We've published a new TechNet guide to many of the changes between Windows Server 2008 and Windows Server 2008 R2; it's definitely worth a look and has good links to more details. This guide is not exhaustive in its beta form (where's the DFSR love?) but it's definitely a good start, especially if you have not been keeping up on Win7 lately. Here's a snippet:

Active Directory Recycle Bin

Information technology (IT) professionals can use Active Directory Recycle Bin to undo an accidental deletion of an Active Directory object. Accidental object deletion causes business downtime. Deleted users cannot log on or access corporate resources. This is the number one cause of Active Directory recovery scenarios. Active Directory Recycle Bin works for both AD DS and Active Directory Lightweight Directory Services (AD LDS) objects. This feature is enabled in AD DS at the Windows Server 2008 R2 forest functional level. For AD LDS, all replicas must be running in a new "application mode." For more information, see What's New in AD DS: Active Directory Recycle Bin.

You can check out the whole guide here: Changes in Functionality from Windows Server 2008 to Windows Server 2008 R2 (Beta)

If you're still living the Win2003 life and need to see what vanilla Win2008 buys you, check out the companion guide here: Changes in Functionality from Windows Server 2003 with SP1 to Windows Server 2008

You can still grab the beta ISO for Win2008 R2 server anytime from here.

And yes - we have big plans for blogging about DS technologies in Windows 7 and R2. Look for these around the Release Candidate timeframe, as we'd like to make sure things are good and stable before we start pumping out info. :)

- Ned 'Thin-Blooded Yankee’ Pyle

PS: Kids always love a snow day…