StdQry

DNS auto-configuration during DCPROMO

karammasri — Wed, 11 Apr 2012 19:10:00 GMT

Have you ever installed the DNS Server service as part of the promotion of a new DC, to end up wondering how this new DNS Server got automatically configured to use forwarders and root-hints? Have you noticed that this process also adds the loopback adapter of the server to the DNS search list in the resolver?

When DCPROMO installs the DNS Server service it also activates, by default, the auto-configuration of the DNS Server service. This auto-configuration process configures the forwarders list, the root-hints and the resolver, among other things, like creating the zones if required.

NOTE: this auto-configuration can be skipped by passing the SkipAutoConfigDns promotion parameter to dcpromo.

I wanted to give the information on how this auto-configuration is done regarding the forwarders list, root-hints and resolver. Remember that you can always edit these parameters after they are configured to match your DNS design.

Auto-configuration of Forwarders

The first step is to configure the forwarders list for the new DNS server. Two possible methods are used to build this list. The second method is skipped if the first one is successful.

Try to copy the forwarders list from a peer DNS server. A peer DNS server is any DNS server that has a copy of this DC domain’s zone. To get the peer server list the process queries for the NS list of the domain’s zone and then contacts each server returned on the list until it finds one from which it can copy the forwarders list. Once the process finds a peer from which it can copy the forwarders list it skips the next step. If no peer is found (because the NS query returned empty), none of them could be contacted, or none of them has forwarders configured, then move to step #2.
If the previous step was not able to provide a forwarders list, then use as forwarders all the DNS Servers that are currently listed in the resolver for all the adapters, without any specific order.
If none of the previous two steps can provide a forwarders list, then the new DNS server will not have forwarders configured.

NOTE: remember that having a DNS server without forwarders is a valid configuration for the service. In this case the server will use its root-hints to resolve names in namespaces that it does not host.

Examples:

Suppose you are promoting a new server in a new domain named contoso.local. The server is pointing to two DNS servers: 10.0.0.1 and 10.0.0.2. None of these servers have the zone contoso.local:

The first method is to get the forwarders list from a peer. The server will issue a query to its DNS servers for the Name Servers list (NS record type) of the zone contoso.local. Because this zone does not exist yet then this query will return empty. As we cannot find a peer to copy the forwarders from we go to method #2.
The forwarders list is built as 10.0.0.1 and 10.0.02 in the second method. This is because we are just copying the DNS servers listed in the resolver.

Suppose you are promoting a new server in an existing domain named contoso.local. The server is pointing to three DNS servers: 10.0.0.1, 10.0.0.2 and 10.0.0.3 which are DCs with DNS for the same domain and have a copy of contoso.local. 10.0.0.1 does not have forwarders configured, 10.0.0.2 is forwarding to 10.1.1.1 and 10.1.1.2 and 10.0.0.3 is forwarding to 10.2.2.1 and 10.2.2.2 (this is a very weird, and probably unstable, DNS configuration but I just wanted to use it as an example):

The query for the NS list of contoso.local will return 10.0.0.1, 10.0.0.2 and 10.0.0.3 (suppose that in this order). We first contact 10.0.0.1 to get its forwarders list, and we find that it is empty, then we contact 10.0.0.2 and query for its forwarders list, and we get 10.1.1.1 and 10.1.1.2. At this point we already got a list, then we skip querying any other server returned by the query (like 10.0.0.3), and also skip method #2.

Suppose you are promoting a new server in a new domain named contoso.local. The server is not pointing to any DNS servers (probably because you are creating a new forest in an environment that does not have DNS yet):

The query for the NS list of contoso.local will return empty, then we cannot use this method to get the forwarders list from a peer.
We cannot create the forwarders list by copying the configuration in the resolver, because there are no DNS servers listed there.
In this case the new DNS server will end-up with no forwarders configured.

Auto-configuration of Root-Hints

The second step is to configure the root-hints for the new DNS server. This is done by querying the root-hints list for all the servers that we have configured in the resolver, and use the list that has the largest number of servers on it; if more than one has the largest number of elements, then we choose the first one that we found with that number. Note that we are not merging the lists that we got from each server, we are just comparing the sizes of the lists and using the one with the largest number of elements on it.

One condition that should be met to accept the root-hints lists is that all the lists should “intersect” in at least one element, i.e., at least one DNS server should be present in all the lists to consider that the lists are valid. If no intersection occurs, then we do not copy any of the lists. This would be a very unusual, and probably wrong, configuration.

Remember that the DNS Server service comes pre-configured with a root-hints list (the Internet root-hints), then even if we cannot copy the list from another server our new DNS will still have a root-hints list to use.

This step is important in environments that have deployed an internal root zone (“.”) because the new DNS server will be auto-configured to use the servers hosting that zone as its root-hints. In the case that this server also hosts the root zone, then the root-hints list will be disabled once the service starts (a server hosting a root zone cannot have root hints).

Auto-configuration of the Resolver

The third step is to configure the resolver (DNS Client) to use the new DNS Server. This is done by adding the loopback address to the DNS search list. The position where the loopback is added is determined by whether we are creating a new forest, new domain, new tree, new replica RWDC or new RODC. The logic is like this:

The new DC is the first DC of the forest: replace the resolver list and set it to point only to the loopback.
The new DC is the first DC of a new tree in an existing forest: add the loopback as the first one on the resolver list. Existing entries in the list are maintained.
The new DC is the first DC of a new child domain in an existing forest: add the loopback as the first one on the resolver list. Existing entries in the list are maintained.
The new DC is an additional RWDC in an existing domain: add the loopback as the last one on the resolver list. Existing entries in the list are maintained.
The new DC is an RODC: add the loopback as the first one on the resolver list. Existing entries in the list are maintained.

Examples:

Suppose you are promoting a new server. The server is pointing to two DNS servers: 10.0.0.1 and 10.0.0.2. After the auto-configuration of the resolver the list will look like:

If the server being promoted is the first DC in a new forest: 127.0.0.1 (the existing list is replaced with the loopback)
If the server being promoted is the first DC in a new tree in an existing forest: 127.0.0.1, 10.0.0.1, 10.0.0.2 (the loopback is added to the beginning of the list)
If the server being promoted is the first DC in a new child domain in an existing forest: 127.0.0.1, 10.0.0.1, 10.0.0.2 (the loopback is added to the beginning of the list)
If the server being promoted is an additional RWDC in an existing domain: 10.0.0.1, 10.0.0.2, 127.0.0.1 (the loopback is appended to the end of the list)
If the server being promoted is an RODC: 127.0.0.1, 10.0.0.1, 10.0.0.2 (the loopback is added to the beginning of the list)

Other auto-configuration Tasks

There are other tasks that are done by the auto-configuration process, like creating the zones when needed. This process is straight forward and well understood, so I will leave it out of this post for now. Let me know in the comments if you would like to read a post about the auto-creation of the zones during dcpromo.

As always, I hope that you find this information helpful.

DHCP Server in DCs and DNS Registrations

karammasri — Tue, 03 Apr 2012 16:35:00 GMT

One common deployment scenario for the DHCP Server service is to have it installed in domain controllers. When this scenario is used it is necessary to define the alternate credentials to be used by DHCP when doing DNS registrations on behalf of the DHCP clients. I wanted to blog about the effects on the registration of the A and PTR records if the DHCP Server is running in a DCs without alternate credentials.

The default DHCP configuration is that clients register their A records and the DHCP Server registers the PTR records. The idea is that the client is the owner of its name, so it is responsible of registering it the DNS namespace, but the owner of the IP is the DHCP server, then it should control its registration in DNS.

This default configuration of this option can be seen in the following screenshot:

The configuration can be done at the server, scope and reservation level. The precedence order is: reservation then scope then server (configuration at the reservation level takes precedence over what is defined at the scope level, which takes precedence over what is configured at the server level). For Windows Server 2003 this dialog can be opened by right-clicking over the server name and choosing properties. For Windows Server 2008 and Windows Server 2008 R2 the dialog can be opened by right-clicking over IPv4 and the selecting properties. For all the versions of the OS, the scope and reservation properties can be opened by right-clicking over them and choosing Properties and changing to the DNS tab.

The alternate credentials to be used by the DHCP Server for the DNS registrations are configured under the Advanced tab of the server properties:

What happens when the DHCP Server service is installed in a DC and no alternate credentials are configured?

A common error is to think that the DHCP Server service running in a DC will use its service account security context to register records in DNS if no alternate credentials are configured, and then there is security risk. In fact, this is not the behavior of the DHCP Server in a DC.

If the DHCP Server service detects that it is running in a domain controller, and no alternate credentials for DNS registrations have been configured, then it decides to not do any registrations for DHCP clients and logs event DHCP/1056.

NOTE: this does not affect other registrations being done by the computer where the DHCP Server service is running, it only affects the registration of DNS records by the DHCP Server on behalf of the DHCP clients.

What is the side-effect of this?

When the DHCP Server decides that it is not going to do registrations for DHCP clients, it stops setting option 81 in the responses to clients (option 81 is used to negotiate who registers what between the DHCP Server and the DHCP Client). If the client does not get this option in the response from the server then it goes and does its own registrations.

Recommendations

If the DHCP Server is configured to run in a DC, make sure that the alternate credentials for DNS registrations are correctly configured.
Use a “normal” user account, not an administrative or privileged account, for the alternate credentials. Just make sure to use the Password Never Expires option. There is not need to add this account to any special group. The steps to configure these credentials are documented in http://support.microsoft.com/kb/282001.
If there are more than one DHCP Server in the environment, try to use the same account for the alternate credentials in all of them.

DNS Clients and Timeouts (part 2)

karammasri — Thu, 15 Dec 2011 01:09:54 GMT

In the first part of this blog post I described the behavior of the DNS client when there are multiple entries in the DNS servers list. In this second part I will try to explain how the Windows DNS Client works when dealing with timeouts and retries.

Note: From now on when I refer to DNS Client I am referring to the Windows implementation of this service. I will also highlight the variances in different versions of Windows when applicable.

How much time is the timeout and how many times do we retry a query?

The DNS client uses an array that defines the timeout to use in each attempt to resolve a query instead of a single fixed value as the same timeout for all the attempts. What this means is that the timeout used during each attempt is not necessarily always the same. Each element of the array defines the timeout to use during each attempt. The query is retried as many times as there are elements in the array, and the process stops when there are no more elements in the array.

For example, suppose that the Timeout array has these values: [1, 1, 2, 4, 4]. The first attempt to resolve a query will timeout after 1 second (first element of the array), the second attempt will timeout after 1 second (second element of the array), the third after 2 seconds, the fourth after 4 seconds, and so on. As there are 5 elements in the array, we will stop retrying after 5 attempts.

The pre-defined timeout array for each of the currently supported versions of Windows are (values shown are in seconds):

	Timeout
OS	[0]	[1]	[2]	[3]	[4]
Windows XP	1	1	2	4	7
Windows Server 2003	1	1	2	4	4
Windows Vista & Windows Server 2008	1	1	2	4	4
Windows 7 & Windows Server 2008 R2	1	1	2	4	4

These timeouts can be customized using the registry value HKLM\System\CurrentControlSet\Services\dnscache\Parameters\DNSQueryTimeouts. This value does not exist by default and then the pre-defined default array just mentioned is used. If the value is defined then it should have a type of REG_MULTI_SZ (multi-line string) with each line containing one value of the array with the last line having a 0 to indicate the end of the list.

If any of the values in a line is higher than 30 then a value of 30 is used for that line instead. If the total sum of the values is higher than 120 (2 minutes) then the list is shortened from bottom-up to remove values until the total timeout is less than 120.

Which DNS Servers do we query in each attempt?

The DNS Client queries the following DNS servers in each attempt:

In the first attempt we query the preferred DNS server of the preferred network adapter only. The preferred DNS server is the first server listed for that adapter. The adapters are sorted based on their binding order and the preferred adapter is the one at the top of the list. You can change this binding order by opening ncpa.cpl and going to the menu Advanced/Advanced Settings…

Adapters that are disabled, disconnected, do not have TCP/IP enabled or have no DNS servers listed are ignored.
In this attempt we query one DNS server only.
If the previous attempt times-out, retry the query with the next best DNS server for all the adapters. The next best DNS server for an adapter is the next on its list that has not already been queried and timed-out.
Note that the lists are managed as circular-lists, once we reach the end of the list for an adapter, the next best server for that adapter will be the first on its list.
In this attempt we query one DNS server per adapter.
If the previous attempt times-out (because none of the DNS servers queried in the previous step answered in the expected time), retry the query with the next best DNS server for all the adapters.
In this attempt we query one DNS server per adapter.
If the previous attempt times-out, retry the query with all the possible DNS servers in all the adapters. This includes even servers that have already timed-out in the previous steps.
In this attempt we query all the DNS servers in all the adapters.
Repeat step (4) until we have run out of attempts. If there are no more attempts then return an error to the caller.
In this attempt we query all the DNS servers in all the adapters.

It is important to clarify something for steps 2 to 5 as multiple servers are queried in those steps: as long as one of the servers queried in that attempt responds, with either a positive or negative answer, then the query is considered resolved. It is OK if the other servers queried do not respond as we already have an answer which is what we wanted.

Using the information about the default Timeout array and the retry logic that was just described we can see that each attempt will timeout, by default, after:

First attempt (step 1): Preferred DNS Server on the preferred adapter: times-out after 1 second
Second attempt (step 2): Next best server for all the adapters: times-out after 1 second
Third attempt (step 3): Next best server for all the adapters: times-out after 2 seconds
Fourth attempt (step 4): All DNS servers in all the adapters: times-out after 4 seconds
Fifth attempt (step 5): Repeat of step (4): times-out after 4 seconds (7 seconds in Windows XP)

After the fifth attempt times-out there are no more elements in the array to use, then we stop the query and return an error to the caller. As our default Timeout array has 5 elements, we try to resolve the query in 5 attempts. The total waiting time before we give up the query is 12 seconds (15 seconds in XP) which is the sum of all the values in the array.

Important note: the DNS servers list is kept in memory by the dnscache service. The next best server is determined based on a priority. All the servers start with the same priority and they are sorted for each adapter based on the precedence in which they were configured. Each time a server times-out its priority is reduced and when a server answers its priority is boosted (error conditions also modify the priority of a server). The next best server for an adapter is the one with the higher priority that is higher in the precedence list (if more than one server have the same priority then the next best is the one that is higher in the precedence list).
It is important to note that this prioritized list is kept across different queries; this means that the priorities are not reset after each query, but they are reused. The idea is that if a server timed-out a recent query then the next query will go to another server with a higher priority first. The effect of this is that the preferred DNS server might not be the first to get the next query if it recently timed-out.
These priorities are reset to the initial default values after an interval named ServerPriorityTimeLimit defined in registry. See http://support.microsoft.com/kb/320760 for more information about this value.
An example of this behavior is a client pointing to two DNS servers: DNS1 and DNS2. The client tries to resolve a name and DNS1 times-out but DNS2 answers. The next query that this client tries to resolve is going to go DNS2 first before being retried in DNS1, because DNS2 would have a higher priority than DNS1.

Making sense of all this information with an example

Suppose we have a computer named CLIENT1 running Windows Server 2003 that has 4 NICs with the binding order NIC1, NIC2, NIC3 and NIC4. The DNS servers list in CLIENT1 is:

NIC1		NIC2		NIC3		NIC4
à	10.110.1.1	à	10.120.1.1	à	10.130.1.1	à	10.140.1.1
	10.110.1.2				10.130.1.2		10.140.1.2
	10.110.1.3				10.130.1.3
	10.110.1.4

The next best server for each adapter is indicated by the “à” symbol. As we are just starting the next best DNS servers for each adapter is the first in their list. Instead of showing how the priorities are modified after each timeout we are going to use a circular list to select the next best server (the effect is the same, and the example looks easier to understand).

We also have a Timeout array in CLIENT1 that looks like this:

Timeout
à	1
	1
	2
	4
	4

The “à” symbol indicates the value to use as the timeout for our next attempt. As we are just starting to resolve a query, we are at the first element of the array.

A process in CLIENT1 needs to resolve a name. Assume that none of the configured DNS servers are reachable so all the attempts time-out.

First attempt: Send query to 10.110.1.1 (best for NIC1 which is the preferred adapter). Wait at most for 1s (the current value in the Timeout array) for an answer.
After this attempt times-out our DNS table and Timeout array will look like this (symbols in red indicate changes from the previous state):

NIC1		NIC2		NIC3		NIC4
	10.110.1.1	à	10.120.1.1	à	10.130.1.1	à	10.140.1.1
à	10.110.1.2				10.130.1.2		10.140.1.2
	10.110.1.3				10.130.1.3
	10.110.1.4

Timeout
	1
à	1
	2
	4
	4

Second attempt (or first retry): Send query to: 10.110.1.2 (next best for NIC1), 10.120.1.1 (next best for NIC2), 10.130.1.1 (next best for NIC3) and 10.140.1.1 (next best for NIC4). Wait at most for 1s (the current value in the Timeout array) for an answer from any of the servers queried.
After this attempt times-out the DNS table and Timeout array will look like this:

NIC1		NIC2		NIC3		NIC4
	10.110.1.1	à	10.120.1.1		10.130.1.1		10.140.1.1
	10.110.1.2			à	10.130.1.2	à	10.140.1.2
à	10.110.1.3				10.130.1.3
	10.110.1.4

Timeout
	1
	1
à	2
	4
	4

Third attempt: Send query to: 10.110.1.3, 10.120.1.1 (NIC2 has just one DNS server listed then this server is always the best server for it), 10.130.1.2 and 10.140.1.2. Wait at most for 2s for an answer from any of the servers queried.
After this attempt times-out the DNS table and Timeout array will look like this:

NIC1		NIC2		NIC3		NIC4
	10.110.1.1	à	10.120.1.1		10.130.1.1	à	10.140.1.1
	10.110.1.2				10.130.1.2		10.140.1.2
	10.110.1.3			à	10.130.1.3
à	10.110.1.4

Timeout
	1
	1
	2
à	4
	4

Fourth attempt: Send query to all DNS servers in all the adapters (including those that timed-out in previous attempts). Wait at most for 4s for an answer from any of the servers queried.
At the end of the waiting time for this attempt the Timeout array will look like this (the DNS list table is not included as we will not use it again for this example):

Timeout
	1
	1
	2
	4
à	4

Fifth attempt: Send query to all DNS servers in all the adapters. Wait at most for 4s for an answer from any of the servers queried.
After this attempt times-out we have run out of values in the Timeout array, then we give up and return an error to the caller.

Where is the Network Trace?

You can see the behavior of the previous example in a network trace:

Frame #3 shows the first attempt: preferred DNS server of the preferred adapter. We use one frame only for this attempt.
Frames #9 to #15 show the second attempt: next best DNS server in all the adapters. Notice how these frames have a time delta of 1s after the first attempt. We have 4 frames because we are querying 1 DNS server for each adapter and we have 4 NICs.
Frames #17 to #23 show the third attempt: next best DNS server in all the adapters. Notice how these frames have a time delta of 1s after the second attempt. We have 4 frames again because we are querying 1 DNS server for each adapter.
Frames #25 to #43 show the fourth attempt: all the DNS servers in all the adapters. Notice how these frames have a time delta of 2s after the third attempt. We have 10 frames because we are querying all the DNS servers in all the adapters, and we have a total of 10 servers to query: 4 for NIC1 + 1 for NIC2 + 3 for NIC3 + 2 for NIC4.
Frames #45 to #63 show the fifth attempt: all the DNS servers in all the adapters. Notice how these frames have a time delta of 4s after the fourth attempt. We have 10 frames for this attempt too.
We have no more frames because we do 5 attempts (remember the Timeout array has 5 elements by default). After the previous attempt times-out, which is going to be 4s (recheck the Timeout array again if you don’t remember where this value of 4s comes from), then we return an error to the caller.

Conclusion

Hopefully after reading this two-part blog post you have a better understanding of how the Windows DNS client works and the logic it follows when it deals with timeouts.

Based on this information, keep in mind these best practices:

Configure the clients to point to more than one DNS server for fault-tolerance. Do not list more than one server to overcome disjoint DNS namespaces, and if you are going to do so, understand the risks and consequences.
Try to have the DNS list in the clients ordered based on the “closeness” (in network terms) to the DNS servers to avoid retries due to timeouts.
Try to have clients use DNS servers that have the information that they are going to query more often; in the case of domain members these would be the DNS servers that have the client domain’s zone.
Maintain an internal DNS infrastructure and hierarchy where names can be resolved independently of the internal DNS server that is queried. For DNS implementations that support multi-domain AD environments, make sure that any DNS servers can resolve any names no matter the domain where the names are registered.
Note: saying that any DNS servers can resolve any names does not mean that all of them have a copy of all the zones in the environment. What it means is that all of them have a way to find the name in the DNS hierarchy because the forwarders/stubs zones/delegations/secondary zones are properly configured.

DNS Clients and Timeouts (Part 1)

karammasri — Fri, 02 Dec 2011 14:18:35 GMT

Some time ago I was helping a customer with a case related to the Windows DNS client. This customer was getting inconsistent name resolution results and he wanted to know why this was happening. The issue that this customer faced was related to the configuration of the DNS servers list in the clients and an incorrect assumption he made about the way that this list is used when a name has to be resolved.

I wanted to start this blog about DNS with a post that tries to clarify some of the concepts related to the use of the DNS servers list and timeouts by the Windows DNS client. In this first part I will describe a sample scenario, a "solution" to a requirement using an incorrect, but common, assumption and the problem with this solution. In the second part I will explain the behavior of the Windows DNS Client when dealing with timeouts.

The sample scenario

The sample environment consists of two Active Directory single-domain forests named contoso.local and fabrikam.local, both in the same location/LAN. These domains are managed by different people, with no trust relationship between them.

contoso.local has two domain controllers: contosodc1.contoso.local with IP 10.24.8.11 and contosodc2.contoso.local with IP 10.24.8.12. There are also two workstations joined to the domain: client1.contoso.local with IP 10.24.9.101 and client2.contoso.local with IP 10.24.9.105.
fabrikam.local has two domain controllers: fabrikamdc1.fabrikam.local with IP 10.24.8.31 and fabrikamdc2.fabrikam.local with IP 10.24.8.32. There is a member server in this domain named server1.fabrikam.local with IP 10.24.8.35.

All the DCs are running the DNS Server service with forwarders configured to the ISP. No name resolution is configured between the two domains. None of the internal domains are published on the Internet DNS servers. WINS is not used.

Members in each domain have their local domain's DCs listed as their DNS servers.

The requirement and the "solution"

Users of CLIENT1 and CLIENT2 need to access a share in SERVER1. The administrators decide to create user accounts for these users in fabrikam.local and have them provide these credentials every time they access this share instead of creating a trust relationship. Also, due to political issues between the administrators, neither of the groups wants to forward DNS queries to the other domain or make any changes to their DNS servers that implied contacting the other domain's DNS servers or listing the other domain's servers locally.

Now when the users in CLIENT1 and CLIENT2 need to connect to server1.fabrikam.local they first need to resolve its name to an IP address. This name resolution is not available in the current environment. The "solution" that the administrators in contoso.local decide to use is to list FABRIKAMDC1 and FABRIKAMDC2 as alternate DNS servers in CLIENT1 and CLIENT2 intermixed with their current DNS servers. The configuration of the DNS servers list for these clients is then going to be: CONTOSODC1, FABRIKAMDC1, CONTOSODC2 and FABRIKAMDC2. The desired configuration would look like this:

This solution is based on the (wrong) assumption that if these clients need to resolve server1.fabrikam.local then the secondary DNS server 10.24.8.31 (FABRIKAMDC1) is going to be used to resolve it because the primary DNS server 10.24.8.11 (CONTOSDC1) will not be able to do so.

Does this solution work?

Luckily (I will explain in a little bit why I am using this word) this solution works sometimes, but most of the times it fails. When it fails the clients are going to get a message that the name of SERVER1 cannot be found.

What is wrong with this solution?

The problem with this solution is the wrong assumption that all the DNS servers in the list are going to be queried until the name is found (positive answer) and that the query will fail only when all of the listed DNS servers answer that the name does not exist (negative answer). In other words: the client will try all the possible DNS servers it has configured until it gets a positive answer before it gives up the query and accepts any negative answer.

So what is the actual behavior of the DNS client?

The actual behavior of the DNS client is that it is going to query its DNS servers in the order that they are listed until an answer, either positive or negative, is received. Once an answer is received, either positive or negative, the DNS client stops the query process and gives that answer back to the calling application. Only when a query to a DNS server times-out (or reports a server error) is when the client retries the query with the next DNS server in the list. In other words: negative answers do no trigger retries with alternate DNS servers, only timeouts (and other errors) do.

Note #1: in the Windows DNS Client this behavior of querying servers in order slightly changes after 3 attempts to resolve a query time-out. I will explain this behavior in part 2 of this post.

Note #2: the calling application might have some logic to retry queries that receive a negative response, but this would be outside the knowledge of the DNS Client process.

Think of this concept as: the DNS client is looking for an answer, not for a positive answer. Once the client gets an answer it "trusts" that the DNS server that sent it did its best to get the correct answer and then it stops querying other servers.

In a DNS infrastructure any server should be able to resolve a name as long as the name exists in the DNS namespace. This means that it does not matter the DNS server that you query because you will eventually get the right answer (positive or negative). Multiple DNS servers are configured for fault-tolerance, not because they have access to disjoint DNS namespaces.

NOTE #3: a similar misconception exists about the use of multiple forwarders, but this will be the topic of another blog post.

So why does this solution work sometimes "by luck"?

This solution works when the query asking for server1.fabrikam.local that is sent to CONTOSDC1 times-out and is retried by the client in the secondary server FABRIKAMDC1. In this case FABRIKAMDC1 will be able to immediately answer with the IP of SERVER1.

What could be the cause for CONTOSODC1 to not answer before the client times-out? CONTOSODC1 has to rely on its forwarders to resolve names in fabrikam.local (remember that CONTOSODC1 does not have a copy of the zone fabrikam.local or any resolution to that domain) and the forwarders could be slow to respond, or the link to the ISP could congested, or the forwarders could be wrongly configured, or the forwarders timeout in CONTOSODC1 were set to high values, and a long etcetera.

This solution breaks when another client sends the same query to CONTOSDC1 shortly after the first query was resolved. This is because CONTOSDC1 would have then already cached a negative response for the name and then it can immediately answer to this query from its cache without contacting the forwarders. The solution also breaks when the name is not in the cache of CONTOSODC1 but the process of querying the forwarders allows it to answer (with a negative response) before the client times-out and switches to FABRIKAMDC1.

At the end for this "solution" to work the client has to be "lucky enough" to be querying CONTOSODC1 when the name is not already in the negative cache of this DNS server and the time to get a response back from the forwarders is higher than the client timeout.

If you are still wondering why CONTOSODC1 will always answer with a negative response for queries in fabrikam.local then you have to remember that neither CONTOSODC1 nor any of the Internet DNS servers have any knowledge of domain fabrikam.local, so none of them have any means to positively resolve names in this domain.

Can you show me some traces?

Here are some network traces that show the behavior that I just described:

In the first trace we have the original configuration of CLIENT1 listing only the DNS servers of its own domain and trying to resolve the name server1.fabrikam.local:

As expected both of the DNS servers answered with a negative response (name error) in packets #3 and #4 because none of them can resolve the name. Notice that CLIENT1 resent the query to CONTOSODC2 in packet #2 because the query to CONTOSODC1 in packet #1 timed-out after 1 second (I will explain this timeout value in Part 2). This timeout was probably due to CONTOSODC1 waiting to get an answer from its forwarders but we can see that it eventually sent back an answer in packet #3 (same logic applies for the delayed response from CONTOSODC2 in packet #4). We will use this fact that CONTOSODC1 takes longer than the client timeout to answer back to the client to analyze the next traces.

The next trace shows what happens when we configure CLIENT1 to use the DNS servers of fabrikam.local as alternate DNS servers using the "solution" we previously discussed (assume that the DNS Server service cache in CONTOSODC1 is empty at this point):

Notice how CLIENT1 first tried to resolve the name using CONTOSODC1 in packet #1; after 1 second it timed-out and retried the query using FABRIKAMDC1 in packet #2 who answered almost instantaneously in packet #3. CLIENT1 then got a positive answer indicating the IP address 10.24.8.35 for server1.fabrikam.local (notice the success answer containing this IP in packet #3). Also notice that some seconds later we have packet #4 which is the (delayed) response from CONTOSODC1 indicating that the name cannot be found, but at this point CLIENT1 has already resolved the name and does not need to take care of this answer.

Now what happens if CLIENT2 (who has the DNS servers list configured exactly as CLIENT1) tries to resolve the same name just after CLIENT1 resolved the name as in the previous trace (packets #1 to #4 from the previous trace are included to reinforce the sequence of events):

Notice how in packet #5 CLIENT2 sends the query to its primary DNS server CONTOSODC1. CONTOSODC1 has already cached that the name does not exist (due to the query that CLIENT1 previously sent) and can almost instantaneously answer back to CLIENT2 with a negative answer in packet #6. Notice how CLIENT2 does not retry the query using its secondary DNS server FABRIKAMDC1 because it already got an answer from the primary server.

Note #4: in a future post I will blog about the use of the positive and negative caches in Microsoft DNS Server.

In the next part of this post I will explain how the Windows DNS client deals with timeouts and its retry logic.

I hope that you find this information useful. Please provide your feedback about the content in the comments sections. I have several ideas for the next posts and I would like to hear your comments about topics that might be interesting for you.