The Case of the Random DFS Access Denial

The Case of the Random DFS Access Denial

  • Comments 3
  • Likes

Hi, this is Bobby from the Microsoft Directory Services support team. Today we are going to discuss some confusing behavior that I’ve observed regarding inconsistent DFS access. While the root of this is issue is easy to remediate, it can be difficult to find the offending servers.

Symptoms

When first presented with this issue, I was asked to help determine the nature of users getting prompted to connect to a DFS root. DFS roots are defined in “How DFS Works as

The starting point of the DFS namespace. The root is often used to refer to the namespace as a whole. A root maps to one or more root targets, each of which corresponds to a shared folder on a separate server. The DFS root must reside on an NTFS volume. A DFS root has one of the following formats: \\ServerName\RootName or \\DomainName\RootName.

The dialog was as follows:

image

Prior to calling Microsoft support, the customer was able to successfully connect to the NETLOGON and SYSVOL share of the domain without issue (\\contoso.com\sysvol and \\contoso.com\netlogon) Rebooting the client machine or having the client logoff would at times alleviate the problem. After a random period of time, users could access the DFS root, would later lose access, and users who did not work before would start working.

Investigating the issue, it was noticed that the "Distributed File System" MMC would change as time passed. For reference, I have included a picture of what it should look like when all is ok.

image

Using the "Distributed File System" MMC to check the status of the root would provide differing results at different times.

image

In this picture, it appears that three of the four root targets are inaccessible. When the MMC was checked later (or the user logged off and then back on) the results changed:

image

When using the “DFS Management” MMC from 2003 R2, The display is different, but the behavior is the same:

image

Initial Troubleshooting

When we began troubleshooting the issue, we started to look at the output of “dir” and dfsutil /pktinfo. This was to test the access to the DFS root and connectivity to the root shares.

image

At this point, each target appeared accessible. This was an unexpected result. I would have expected that we would get access denied to all of the targets. If we could reach each target, why were getting random “access is denied” when attempting to get a directory listing of the root?

We used the dfsutil /pktflush command to purge the referral cache thus forcing a request for new referrals. Then we ran the dir commands again that returned different results:

image

Using Netmon, we took a trace to see what we can find.

The first thing we attempted was to request a DFS referral for \\contoso.com\root

image

We were returned four targets for the root.

image

This was represented in the pktinfo as follows:

image

At this point we attempted to create a connection to the first server returned in the referral.

image

However we were returned: Status_access_denied.

Normal troubleshooting would be to attempt to connect to the share \\2003-02\root. In this case, we executed dir /b \\2003-02\root.

When we attempt to do this, we received more unexpected results:

image

I would have actually expected this to get an “access is denied” too.

Looking further in the network trace, we noticed that we received a new referral when we attempted to connect to 2003-02 server directly. To illustrate this, I have several screenshots of the referral network traffic.

image

The DFS service returned the following (note that ‘TargetPath: Index:1’ is not 2003-02 but another server):

image

Again the pktinfo:

When connecting to the path \\2003-02\root, we were instead accessing the root share on 2003-01, as detailed in the trace:

image

Cause and Remediation

When connecting to a root DFS share, the DFS service returns referrals for all of the participants on the root target set. So when we executed the command dir \\2003-dc1\root, the DFS service on 2003-dc1 then responded with a new DFS root referral for the path \\2003-dc1\root.  This behavior accounts for the entries in the referral cache for paths to each root (i.e. \\2003-dc1\root, \\2003-01\root, \\2003-02\root, \\2003-03\root). DFS referrals are retuned in a random order for servers in the site of the client. This will cause differencing results for each time the client requests a referral.

The root cause of the issue was actually incorrect share permissions on the “root” share of 2003-02. This can be observed locally using net share sharename

image

Comparing this to the same output of 2003-01, we see the problem:

image

In small environments, connecting to the server and examining the share permissions is an acceptable method, however in large environments this can prove cumbersome. I prefer to use subinacl to verify the permissions remotely. (Subinacl can be downloaded here: http://www.microsoft.com/downloads/details.aspx?FamilyID=E8BA3E56-D8FE-4A91-93CF-ED6985E3927B&displaylang=en)

image

To fix this we could connect to the server and add the permissions. Or it could be altered remotely with the following:. The syntax is : subinacl /share \\2003-02\root /grant=everyone=r

Note: Read is the default share permission in 2003 and later. Your permissions may be different based upon your business needs.

Using DFSGUI.MSC to check the status of the DFS root targets has been deprecated and is no longer available in 2008. This however does not change the behavior of the referrals that are returned.

Using DFSdiag.exe from Windows 2008 does not return any additional information as to the root cause.

Further reading

For more information about the technologies discussed here, please refer to “How DFS Works” http://technet.microsoft.com/en-us/library/cc782417(WS.10).aspx

- Bobby ‘Bucket Head’ McMillan
  • PingBack from http://proteinmyosin.blog-giant.com/2009/05/27/social-worker-msw-robinson-visiting-nurse-andamp-hospice-per-diem/

  • Hi, The Subinacl doc [subinacl /help /full] mentions it does not support DFS volumes.

    While I can't see why Subinacl would not change file even if they are access through DFS links, I can't see what "DFS volumes" means in that case.

    Can you clarify, or confirm that indeed Subinacl works on files accessed through DFS links

    Thanks

  • Hi Thelemanu,

    The tool has hard-coded checks to prevent it from working over DFS paths. I have no idea why though (lost in the mists of time). If you try to do so it will return "DfsPath is not supported". If you point to the 'real' paths - the DFS link target UNC paths - it will work.

    But icacls.exe will work fine and that's available as a download and incldued in Vista/2008+.