Chicken Soup for the Techie

Netmon's view of Kerberos communication, when accessing resources across domains in the same forest.

Domain setup: 

Both Child1 and Child2 are in the same forest with the same parent domain R2dom.local.

Administrator of the Child domain (CHILD1) login to a member server (CH1-Mem) in CHILD1 domain.

After login in the user tries to access \\r2dom-ch2-Mem1 . R2dom-ch2-Mem1 is a member server in Child2 domain.

--> I have used Network monitor to analyze and understand how Kerberos authentication would work, when accessing resource across domain.

Below you see that the Administrator is getting the required Kerberos tickets when accessing resources across domain.

10.2.1.2   = DC of CHILD1.R2DOM.LOCAL

10.1.1.2   = DC of R2DOM.LOCAL

10.3.1.1   = Dc of CHILD2.R2DOM.LOCAL

10.10.10.1   = CH1-Mem in CHILD1

10.10.10.1   10.2.1.2   KerberosV5:TGS Request Realm: CHILD1.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1

                                                                   --> KrbApReq: KRB_AP_REQ (14)

                                                                Ticket: Realm: CHILD1.R2DOM.LOCAL, Sname: krbtgt/CHILD1.R2DOM.LOCAL

** Here you see a Kerberos TGS request being sent to the local domain (CHILD1.R2DOM.LOCAL) DCs for a SPN cifs/r2dom-ch2-Mem1. Local domain TGT sent in the TGS request.

10.2.1.2   10.10.10.1   KerberosV5:TGS Response Cname: Administrator

                                                --> Ticket: Realm: CHILD1.R2DOM.LOCAL, Sname: krbtgt/R2DOM.LOCAL

** Here the local domain DC returns a TGT of the Parent domain R2DOM.LOCAL. This is like a referral being sent to the client as the local domain does not have the right to issue a Kerberos Ticket for cifs/r2dom-ch2-Mem1.

10.10.10.1   10.1.1.2   KerberosV5:TGS Request Realm: R2DOM.LOCAL Sname: krbtgt/CHILD2.R2DOM.LOCAL

--> KrbApReq: KRB_AP_REQ (14)

                                                                Ticket: Realm: CHILD1.R2DOM.LOCAL, Sname: krbtgt/R2DOM.LOCAL

** Now the client machine send a TGS request to the Parent domain R2DOM.LOCAL, requesting for a TGT of another of its Child domain where the cifs/r2dom-ch2-Mem1 resides. When sending this request to the parent domain, the client uses the TGT of the Parent domain received in the earlier referral from local DC.

10.1.1.2   10.10.10.1   KerberosV5:TGS Response Cname: Administrator

                                                --> Ticket: Realm: R2DOM.LOCAL, Sname: krbtgt/CHILD2.R2DOM.LOCAL

**Parent domain sends the TGT of the child2 domain to the client. This can also be taken as a referral to the CHILD2 domain.

10.10.10.1   10.3.1.1   KerberosV5:TGS Request Realm: CHILD2.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1

                                                -->KrbApReq: KRB_AP_REQ (14)

                                                                Ticket: Realm: R2DOM.LOCAL, Sname: krbtgt/CHILD2.R2DOM.LOCAL

** Eventually the Child sends a TGS request for the SPN cifs/r2dom-ch2-Mem1 to the DCs in domain CHILD2.R2DOM.LOCAL who is authorized to issue a ticket for the server (r2dom-ch2-Mem1) in its domain. This time the client uses the CHILD2 domains TGT to make request.

10.3.1.1   10.10.10.1   KerberosV5:TGS Response Cname: Administrator

                                                --> Ticket: Realm: CHILD2.R2DOM.LOCAL, Sname: cifs/r2dom-ch2-Mem1

** Finally the client gets the Kerberos Ticket for  cifs/r2dom-ch2-Mem1, which will help the Administrator user access the shares on r2dom-ch2-Mem1 in domain CHILD2.R2DOM.LOCAL.

The workstation the user 'Administrator" is using to access the resource across domain, also needs a similar ticket.

Tickets for r2dom-ch2-Mem1 is requested from local domain, who returns the error KDC_ERR_S_PRINCIPAL_UNKNOWN, as the r2dom-ch2-Mem1 computer is not a part of the local domain (CHILD1.R2DOM.LOCAL)

10.10.10.1   10.2.1.2   KerberosV5  KerberosV5:TGS Request Realm: CHILD1.R2DOM.LOCAL Sname: r2dom-ch2-Mem1

10.2.1.2   10.10.10.1   KerberosV5  KerberosV5:KRB_ERROR  - KDC_ERR_S_PRINCIPAL_UNKNOWN (7)

Below you see that the client computer is getting the required Kerberos tickets when accessing resources across domain.

10.10.10.1   10.2.1.2   KerberosV5  KerberosV5:TGS Request Realm: CHILD1.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1

10.2.1.2   10.10.10.1   KerberosV5  KerberosV5:TGS Response Cname: CH1-Mem$

10.10.10.1   10.1.1.2   KerberosV5  KerberosV5:TGS Request Realm: R2DOM.LOCAL Sname: krbtgt/CHILD2.R2DOM.LOCAL

10.1.1.2   10.10.10.1   KerberosV5  KerberosV5:TGS Response Cname: CH1-Mem$

10.10.10.1   10.3.1.1   KerberosV5  KerberosV5:TGS Request Realm: CHILD2.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1

10.3.1.1   10.10.10.1   KerberosV5  KerberosV5:TGS Response Cname: CH1-Mem$

-       Abizer

Should IIS be installed on Domain Controller

I have come across various scanarios where System Administrators have installed IIS on Domain Controllers. They do it to efffectively utilize that server hardware, to cut down cost by preventing a need for another server for IIS, some application that needs to be installed on the DC requires IIS etc.

Microsoft does NOT recommend IIS on a Domain Controller running Active Directory. There are 2 mains reasons behind this stand.

1.     By installing IIS on a DC, we will end up increasing the surface attack area on that DC, hence causing a threat to the security database of the domain. This may also effect the servers performance and reliability.

2.     IIS would NOT work correctly as it mainly works with local users and groups which will now become domain users /groups. This would cause permission issues if the ACLs set on different IIS folders and Metabase is not updated correctly.

Below are a few articles that would give you some idea of the issues faced when IIS is installed on a Domain controller.

DCPROMO does not retain permissions on some IIS folders
http://support.microsoft.com/default.aspx?scid=kb;EN-US;332097

 

How To Promote a Member Server Running IIS to a Domain Controller Running IIS
http://support.microsoft.com/kb/300432

 

FIX: ASP.NET does not work with the default ASPNET account on a domain controller
http://support.microsoft.com/default.aspx?scid=kb;EN-US;315158

 

Cannot install a Systems Management Server 2003 Management Point role on Windows Server 2003 domain controllers
http://support.microsoft.com/default.aspx?scid=kb;EN-US;886213

 

Avoid installing IIS on a domain controller
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/3a0742c4-f45a-4504-a232-83dd085bcfb3.mspx?mfr=true

 

Part from the above resources, I strongly recommend viewing the webcast below as it will talk in detail on why IIS is not recommended on DC. This webcast also covers the issues you may face when you install IIS on DC and how it deal with it.

TechNet Webcast: Successfully Running IIS on a Domain Controller - Level 200
http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032245355&Culture=en-US

To summarize, we don’t recommend installing IIS on DCs, but if you have to install it you need to be prepared to deal with issue related to IIS.

- Abizer

Error: "The parameter is incorrect" when connecting to a server using WMI.

You test WMI connectivity remotely using WBEMTEST > Error: "The parameter is incorrect"

Analysis:

Network trace during the issue shows that communication is happening with TCP Port 135 but after that secondary connection other DCOM/WMI interface not happening on other DYNAMIC RPC ports (above 1024).
All ports between the client and the target server are open.

Network trace during the problem: communication only with TCP port 135

10.171.72.119    abizerh-lab  MSRPC   MSRPC:c/o Bind:  UUID{000001A0-0000-0000-C000-000000000046} IRemoteSCMActivator(DCOM)  Call=0x4  Assoc Grp=0x0  Xmit=0x16D0  abizerh-lab   10.171.72.119  MSRPC   MSRPC:c/o Bind Ack:  Call=0x4  Assoc Grp=0x71F7  Xmit=0x16D0  Recv=0x16D0
10.171.72.119    abizerh-lab  MSRPC   MSRPC:c/o Alter Cont:  UUID{000001A0-0000-0000-C000-000000000046} IRemoteSCMActivator(DCOM)  Call=0x4
abizerh-lab   10.171.72.119 MSRPC   MSRPC:c/o Alter Cont Resp:  Call=0x4  Assoc Grp=0x71F7  Xmit=0x16D0  Recv=0x16D0
10.171.72.119    abizerh-lab  DCOM   DCOM:RemoteCreateInstance Request, DCOM Version=5.7  Causality Id={F756624A-7CA5-4534-9F62-6638201C68CD}
abizerh-lab   10.171.72.119 DCOM   DCOM:RemoteCreateInstance Response, ORPCFLOCAL - Local call to this computer
                                                                              ----->ReturnValue: 0x00000057 - ERROR_INVALID_PARAMETER - The parameter is incorrect.

Working trace: You can see that apart from the connections to TCP 135, secondary connection are being made to other UUID of DCOM / WMI interface.

10.171.72.119   Abizerh-lab MSRPC   MSRPC:c/o Request: unknown   Call=0x3  Opnum=0x5  Context=0x0  Hint=0x0
Abizerh-lab    10.171.72.119 MSRPC   MSRPC:c/o Response: unknown   Call=0x3  Context=0x0  Hint=0xE4  Cancels=0x0
10.171.72.119   Abizerh-lab  DCOM   DCOM:RemoteCreateInstance Request, DCOM Version=5.7  Causality Id={03728AE5-CD86-4477-BA31-7B275C0A7CFF}
Abizerh-lab   10.171.72.119 DCOM   DCOM: Response, ORPCFLOCAL - Local call to this computer, Unknown IRemoteSCMActivator Method opnum=0
10.171.72.119   Abizerh-lab  MSRPC   MSRPC:c/o Bind:  UUID{00000143-0000-0000-C000-000000000046} IRemUnknown2(DCOM)  Call=0x1  Assoc Grp=0x0  Xmit=0x16D0 
Abizerh-lab   10.171.72.119 MSRPC   MSRPC:c/o Bind Ack:  Call=0x1  Assoc Grp=0x7688  Xmit=0x16D0  Recv=0x16D0
10.171.72.119   Abizerh-lab MSRPC   MSRPC:c/o Auth3:  Call=0x1
10.171.72.119   Abizerh-lab DCOM   DCOM:IRemUnknown2:RemQueryInterface Request, DCOM Version=5.7  Causality Id={03728AE5-CD86-4477-BA31-7B275C0A7CFF}
Abizerh-lab   10.171.72.119 DCOM   DCOM:IRemUnknown2:RemQueryInterface Response, ORPCFNULL - No additional information in this packet
10.171.72.119   Abizerh-lab MSRPC   MSRPC:c/o Alter Cont:  UUID{D4781CD6-E5D3-44DF-AD94-930EFE48A887} IWbemLoginClientID(WMIRP)  Call=0x2
Abizerh-lab   10.171.72.119 MSRPC   MSRPC:c/o Alter Cont Resp:  Call=0x2  Assoc Grp=0x7688  Xmit=0x16D0  Recv=0x16D0

Dumped the endpoint mapper database of the TARGET server (we are trying to connect to via WMI) containing the details of the services asociation with different protocols/ports.
- RPCDUMP /s target_server /v /i > rpcdump.txt
**RPCDUMP is a part of windows resource kit.

This output during the problem, from the target server, showed services listening on ncacn_np, ncalrpc but NO services listening on ncacn_ip_TCP. In a normal scenario, the above output should show atleast a few services listening on Dynamically allocated TCP ports i.e. associated with ncacn_ip_TCP protocol. A few services that you should fine listening on all Windows server is SAM {12345778_1234_abcd_ef00_0123456789ac} or SVCCTL {367abb81_9844_35f1_ad32_98f038001003}.

You should see something like this in the RPC dump to confirm that Dynamic allocation of TCP ports is happening on the server.

 ProtSeq:ncacn_ip_tcp
 Endpoint:5003
 NetOpt:
 Annotation:
 IsListening:YES
 StringBinding:ncacn_ip_tcp:ABIZERH-LAB[5003]
 UUID:12345778-1234-abcd-ef00-0123456789ac
 ComTimeOutValue:RPC_C_BINDING_DEFAULT_TIMEOUT
 VersMajor 1  VersMinor 0

After receiving the hint from the RPCdump output, of the Target server, we looked up the following registry key/values on the Target server.

 HKEY_LOCAL_MACHINE\Software\Microsoft\Rpc\Internet
 - Ports
 - PortsInternetAvailable
 - UseInternetPorts

**These values were present and configured on the Target server.

Note: the above registry values are not present by default and are set it case you want to set a range of TCP ports for RPC dynamic port allocation.

How to configure RPC dynamic port allocation to work with firewalls
http://support.microsoft.com/kb/154596

So was this RPC dynamic port allocation configuration causing the problem? NO NO NO

On the problem target server, we found that the RPC dynamic port allocation configuration were not set correctly i.e. the Port range had a space between them for example Ports = '5100 - 5200' instead of '5100-5200'. This was basically confusing the OS where it knew that the RPC dynamic allocation restriction was set but could make sence of the values, hence causing it not to allocate any ports to any services at all.

Troubleshooting the error "Not enough storage is available to complete this operation"

I have come across a few issues where I have seen the above error. Below are two scenarios of the issue and the symptoms that I've noticed during that time.

·         Domain Workstations going into a state where they are unable to access resources over the network.

·         Member Servers unable to access network resources and stop allowing domain users to login.

**In both the above scenarios, the issue seemed to get fixed for some time if the computer was Rebooted. Further investigations show a common set of errors being reported on these problem computers.

5719       NETLOGON
This computer was not able to set up a secure session with a domain controller in domain %Domainname% due to the following:   Not enough storage is available to process this command. This may lead to authentication problems. Make sure that this computer is connected to the network.

40960    LSASRV
The Security System detected an authentication error for the server cifs/hostname.domain,com.  The failure code from authentication protocol Kerberos was "There are currently no logon servers available to service the logon request.   (0xc000005e)".

1053       Userenv
Windows cannot determine the user or computer name. (Not enough storage is available to complete this operation. ). Group Policy processing aborted.

Though this errors clearly says "Not enough storage.." you may still see enough available RAM and good amount of free space on the hard drive, on that problem computer. So why does it still say "Not enough storage is available to complete this operation"?

What I understood from this error was that a RESTRICTED amount of MEMORY allocated for particular type of OS RESOURCE, has got EXHAUSTED.

In both the cases above, the resource type that was getting exhausted or causing other resources to be exhausted was a HANDLE. A Handle is an OS object used to get a reference of another object (like File, registry, port etc.). It’s using these handles that a process will work with open files, read registry keys or work with a network port.

In both the above scenarios, we saw a process (Non-Microsoft) consuming large amount of Handles. You can view the handles owned by different processes through "Task Manager" in Windows. In the "Task Manager", under the Process tab you can add an additional column to display Handles information. Stopping or killing this process that was using large amount of handles restored this computer back from the problem state and every thing started working fine.

Though the problem process/application has to be debugged by its developers but what can help them further is to know what type of handles did the application have the most and to which objects. This can be found easily by using a tool called Handle.exe (http://technet.microsoft.com/en-us/sysinternals/bb896655.aspx). "Handle -a > handle_output.txt" would dump information about all open handles on the computer, sorted by processes into the text file handle_output.txt. You can review the handles under the problem process to check, what type of handle this process has the most.

One of the most common questions being asked here is "how does one conclude which process's handle usage is abnormal?”

The best way to find that out is to check the handles owned by different process from Task Manager and find the process that owns the maximum number of handles. Then check what the process does and if it is normal for that process to be holding references to those many objects via handles.
For example:

·         On a Domain controller you would see LSASS using most amount of handles, this may be normal as LSASS is the main process on a DC which is responsible for most of the work done by that server.

·         On a SQL server you would see the SQL server process having a lot of handles, this may again be quite normal depending on the size of the database the SQL server has and the amount of queries/request coming to it.

·         BUT if an application or a management agent process is using large number of handles like 30,000 or 140,000, it is something that needs to be investigated.

**Another thing to note is, if the handle usage goes down when this process is not actively working or does the handle count keeps increasing. If you see the handle count keeps increasing and never coming down, this may indicate a handle leak, which means the process is requesting and acquiring a handle but not releasing it when its work is done.

There could be other reasons behind the "Not enough storage is available to complete this operation" error too, so just don’t restrict you troubleshooting to the above steps.

Troubleshooting “RPC server is unavailable” error, reported in failing AD replication scenario.

In this scenario when are troubleshooting AD replication between 2 DCs separated by a firewall.

In order to ensure that the important well-known ports required in a domain environment are open on the firewall between these DCs, use the PortqryUI tool.

PortqryUI

http://www.microsoft.com/downloads/details.aspx?FamilyID=8355e537-1ea6-4569-aabb-f248f4bd91d0&displaylang=en

Run this tool on both these DCs to test the communication on a selected set of ports on the target DC (replication partner).

·         Invoke PortqryUI.exe

·         Enter replication partner’s IP or FQDN in the “Destination to query” textbox.

·         Select Pre-defined query – “Domains and Trusts”.

·         Hit the “Query” button and let it finish.

·         Save the above output to a text file.

PortqryUI

Go through the PortqryUI query result by searching for “Return Code” phrase in the output.

è  If the return code is 0, it indicates that this DC was able to communicate with its partner DC on that particular port.

è  The return code 2 is normally reported for UDP ports as we don’t get an ACK for that communication. This can be ignored if it’s returned for a UDP port.

è  The return code of 1 indicates that this DC was unable to talk to the target DC on the respective port. This either indicates that the service related to this port is not running on the target or that port is FILTERED on the firewall.

è  Any other return code also needs investigation.

Sample output of PortqyrUI

In our scenario, we need to ensure that the following ports are open on both these DCs.

·         TCP 135 – Endpoint mapper

·         TCP,UDP 389 - LDAP

·         TCP, UDP 88 - Kerberos

·         TCP 445 - SMB

·         TCP 139 – SMB, Namepipe

·         TCP, UDP 53 – if these servers are DNS servers too.

Out of the above ports, the one that is most IMP to look at in the RPC related errors is TCP 135.

à This is the Endpoint Mapper port. A DC would first communicate with its partner on port 135 to get the details of the TCP ports the NTDS and Netlogon services are listening on. It’s only when it gets this response from the Endpoint mapper that it would communicate with the NTDS (DRS) and Netlogon service on the target DC (Partner DC).

To get the list of the Endpoints on the partner DC and get the list of services and the ports associated with it, we can use another tool called RPCdump. This tool also has the capability of checking if source server can communicate with all endpoint on the destination server.

RPCdump is a part of Windows resource kit.

http://www.microsoft.com/downloads/details.aspx?familyid=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en

Command : RPCDUMP /s destination_server /v /i > RPCdump_destination.txt

In our scenario, we need to review the TCP port the DRS Interface is listening on. You can either search using the phrase DRS or using the following UUID e3514235-4b06-11d1-ab04-00c04fc2dcd2.

Other UUIDs:

e3514235_4b06_11d1_ab04_00c04fc2dcd2        DRS

12345778_1234_abcd_ef00_0123456789ab         LSA

12345678_1234_abcd_ef00_01234567cffb          NETLOGON

12345778_1234_abcd_ef00_0123456789ac         SAM

When looking at this log we need to check if for the respective UUID and IsListening result.

è  IsListening:YES – this means that the source server was able to communicate with target server on the respective port.

è  IsListening:NO – means that this port is filtered on the firewall.

è  IsListening:Unknown – this may mean that you need to investigate further as packets targeted to the respective port may not be reaching the server. In this scenario a simultaneous network trace may help.

Sample RPCdump output

Yon can confirm if the source server can communicate to the destination server on a particular port by using PortqryUI again. This time specify a Port instead of a predefined query.

If you find that the DRS and Netlogon service ports cannot be communicated to, from either of the 2 DCs which are suppose to replicate with each other. Then we should have the network team analyse the Firewall/network device - to allow communication on this port.

In some scenarios, you will see that the above 2 test pass and the DCs are able to communicate with each other on the required ports, but then too the AD replication fails with RPC server unavailable message.

In this scenario, we need to install Network Monitor on both the source and target DCs.

·         Start network monitor capture on both these DCs simultaneously.

·         Force AD replication between these DCs using “AD sites and services” snap-in.

·         Leave the network monitor for 2 to 3 mins after initiating the replication and then stop it.

**The main thing to analyse in this network trace (for RPC errors) is to check if any packets between these DCs are getting dropped. This can be done by looking at the communication between these 2 DCs, in both the simultaneous network traces. If there are packets visible on 1 trace which is not reflected in the other trace, it would indicate that the packet may have got dropped.

I have seen a few firewalls with “Intrusion Prevention System” drop selective packets.

Network Monitor

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=983b941d-06cb-4658-b7f6-3088333d062f

The above troubleshooting should be able to indicate the reason behind the RPC server unavailable messages for sure.

More Information - related to configuration in the above scenario:

Windows 7 - Applocker

Windows AppLocker is a new feature in Windows 7 and Windows Server 2008 R2 is an alternative to the Software Restriction Policies feature.

New with AppLocker

==================

·         Define rules based on file attributes derived from the digital signature, including the publisher, product name, file name, and file version. For example, you can create rules based on the publisher and file version attributes that are persistent through updates, or you can create rules that target a specific version of a file.

·         Rule can both allow or deny access to a file/file types.

·         Assign a rule to a security group or an individual user.

·         Create exceptions for .exe files. For example, you can create a rule that allows all Windows processes to run except Regedit.exe.

·         Use audit-only mode to identify files that would not be allowed to run if the policy were in effect.

·         Applies to Windows 2008 and Windows 7 only.

**Applocker exist along with the old Software Restriction Policy.

**Publisher is a new option with Applocker which is not present in S/W Restriction policy.

AppLocker Step-by-Step Guide
http://technet.microsoft.com/en-us/library/dd723686(WS.10).aspx

-Abizer

Preventing Unwanted/Accidental deletions and Restore deleted objects in Active Directory