In this post, I want to talk about how I troubleshooted a direct access connectivity problem hoping that it might also help you in troubleshooting similar problems.

The issue was that the IPHTTPS client cannot connect to UAG DA server. We first checked static logs collected from a problem client (GPO output, firewall outputs, etc) and all seemed to be fine:

 

- Required services were running on the client (Windows Firewall/IKE and AuthIP/IP Helper)

- Firewall was chosing the correct firewall (Public profile):

 

-------------------------------------

netsh advfirewall show currentprofile

-------------------------------------

 

Public Profile Settings:

----------------------------------------------------------------------

State                                 ON

Firewall Policy                       BlockInbound,AllowOutbound

LocalFirewallRules                    N/A (GPO-store only)

LocalConSecRules                      N/A (GPO-store only)

InboundUserNotification               Enable

RemoteManagement                      Disable

UnicastResponseToMulticast            Enable

 

Logging:

LogAllowedConnections                 Disable

LogDroppedConnections                 Disable

FileName                              C:\Windows\System32\LogFiles\Firewall\pfirewall.log

MaxFileSize                           4096

 

Ok.

 

 - All categories including connection security was owned by Windows Firewall:

 

Categories:

BootTimeRuleCategory                  Windows Firewall

FirewallRuleCategory                  Windows Firewall

StealthRuleCategory                   Windows Firewall

ConSecRuleRuleCategory                Windows Firewall

 

- gpresult output collected confirmed that the client was getting the required policy which is UAG DirectAccess: Clients (UAGServer-FQDN)

 

Then I asked our customer to implement the below action plan to collect dynamic data while reproducing the problem

 

Action plan for log collection:

=========================

The outline of the action plan is below:

 

- Start WFP/DirectAccess/network traces on the DA client

- Start WFP/DirectAccess/network traces on the UAG DA server

- Stop and restart the relevant services (like IKE/ip helper) to trigger the tunnel establishment and see things from scratch

- Once we observe that the client cannot access any internal resources we’ll stop logs on the DA client and UAG DA server

 

1) Please install Network Monitor 3.4 on a problem DA client and on the UAG DA server. You can download the tool at the following link:

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=983b941d-06cb-4658-b7f6-3088333d062f     

 

2) Please disable and stop “IKE and AuthIP IPSec Keying Modules” service with the below commands on the DA client:

Note: All commands need to be run from an elevated command prompt.

 

a) We first disable the service:

 

sc config ikeext start= disabled

(Note: There must be a SPACE between “=” and “disabled”, otherwise the command doesn’t work)

 

b) We stop the service:

net stop ikeext

 

3) Now we start the network capture on the DA client and on the UAG server: (from an elevated command prompt)

 

=> On the DA client:

 

nmcap /network * /capture /file DAClient.chn:100M

 

=> On the UAG DA server:

 

nmcap /network * /capture /file DAServer.chn:100M

 

Note: You have to run the nmcap command from an elevated command prompt on Windows Vista/Windows 7/Windows 2008/Windows 2008 R2

Note: nmcap will create a new capture file once the first one if full (200 MB) and so on. So please make sure that you have enough free disk space on the related drive.

Note: If you get the error message "'nmcap' is not recognized as an internal or external command, operable program or batch file", you may need to run the nmcap command inside "%ProgramFiles%\Microsoft Network Monitor 3" folder

Note: Network trace could be stopped any time by pressing Ctrl + C

 

4) Let’s start tracing for various components at a different elevated command prompt on the DA client and on the UAG DA server:

 

=> On the DA client:

 

netsh wfp capture start

 

netsh trace start scenario=directaccess provider=microsoft-windows-iphlpsvc level=5 capture=yes tracefile=darepro.etl maxsize=350

 

=> On the UAG DA server:

 

netsh wfp capture start

 

netsh trace start provider=Microsoft-Windows-DNS-Client provider=Microsoft-Windows-Iphlpsvc-Trace provider=Microsoft-Windows-WFP provider=Microsoft-Windows-NCSI provider=Microsoft-Windows-NlaSvc provider=Microsoft-Windows-NetworkProfile provider=Microsoft-Windows-WinINet provider=Microsoft-Windows-WinHttp provider=Microsoft-Windows-WebIO provider="Microsoft-Windows-Windows Firewall With Advanced Security" tracefile=c:\logs\UAGDAServer.etl maxsize=400

 

5) Now we’re going to reproduce the problem from the DA client side.We’re going to stop iphlpsvc on the DA client: (the service which implements the IPv6 transition technologies)

 

net stop iphlpsvc

 

6) Just before we start re-enabling the relevant services on the DA client, we run the following ping on the DA client: (with 22 bytes ping)

 

ping -l 22 -n 2 IP-address-of-default-gateway-configured-on-the-client

 

Note: Please replace IP-address-of-default-gateway-configured-on-the-client  with the real default gateway address configured on the client

 

7) Now please run the following commands in the given order on the DA client:

 

net start iphlpsvc

sc config ikeext start= auto

net start ikeext

 

(Note: There must be a SPACE between “=” and “disabled” in "sc config" command, otherwise the command doesn’t work)

 

8) Let’s please wait for 30 seconds or so and then try the following commands on the DA client:

 

ping -l 33 an-internal-server1

 

ping -l 44 an-internal-server2

 

net view  \\an-internal-server1

 

net view  \\an-internal-server2  

 

As an additional test, you can try to reach an internal web server/sharepoint portal etc if any exists from Internet explorer on the DA client (like accessing http://internalserver3/portal)

 

NOTE: Please replace the “an-internal-server” with real ones

NOTE: Please write down the full commands that you run

NOTE: Please take a screenshot of each access attempts

 

9) After you get errors for the above commands, please run the following commands (all from an elevated command prompt) on the DA client and on the UAG DA server to stop the logs running:

 

=> On the UAG DA server:

 

a) To stop WFP tracing:

netsh wfp capture stop

 

b) To stop Directaccess tracing:

netsh trace stop

 

c) To stop Network trace on the UAG DA server:

Please hit CTRL+C at the command prompt where nmcap tool runs

 

=> On the DA client:

 

a) To mark the end of problem reproduce (with 55 bytes ping)

 

ping -l 55 -n 2 IP-address-of-default-gateway-configured-on-the-client

 

b) To stop WFP tracing:

netsh wfp capture stop

 

c) To stop Directaccess tracing:

netsh trace stop

 

d) To stop Network trace on the DA client:

Please hit CTRL+C at the command prompt where nmcap tool runs

 

ANALYSIS 1:

==========

After analyzing the logs collected as per the above action plan, I got the following findings:

 

=> IPHTTPS interafce was not connected with the error number 0x800b0109

 

Interface IPHTTPSInterface (Group Policy)  Parameters

------------------------------------------------------------

Role                       : client

URL                        : https://UAGDAServer-FQDN:443/IPHTTPS

Last Error Code            : 0x800b0109

Interface Status           : failed to connect to the IPHTTPS server. Waiting to reconnect

 

 

err 0x800b0109

# for hex 0x800b0109 / decimal -2146762487 :

  CERT_E_UNTRUSTEDROOT                                          winerror.h

# A certificate chain processed, but terminated in a root

# certificate which is not trusted by the trust provider.

# 1 matches found for "0x800b0109"

 

=> IPHTTPS client doesn’t accept the certificate provided because it doesn’t trust the issuer

 

=> When I checked the network activity, I saw the following communications:

 

2084        15:13:43.441664     463.087187             0.002359 192.168.99.5           192.168.99.100       DNS         Standard query A UAGDAServer-FQDN

2085        15:13:43.442251     463.087774             0.000587 192.168.99.100       192.168.99.5           DNS         Standard query response A 1.1.1.2

 

Now client tries to establish the TLS tunnel:

 

No.     Time            Time since  Delta       Source                Destination           Protocol Identification S-Port     D-Port     Sequence   TCP Len    TCP Ack    TCP Win    Info

   2086 15:13:43.443150 463.088673  0.000000    192.168.99.5          1.1.1.2       TCP      0x58ba (22714) 57134      443        1683802426 0                     8192       57134 > 443 [SYN] Seq=1683802426 Win=8192 [TCP CHECKSUM INCORRECT] Len=0 MSS=1460 WS=256 SACK_PERM=1

   2087 15:13:43.479192 463.124715  0.036042    1.1.1.2       192.168.99.5          TCP      0x61f4 (25076) 443        57134      536292141  0          1683802427 8192       443 > 57134 [SYN, ACK] Seq=536292141 Ack=1683802427 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1

   2088 15:13:43.479356 463.124879  0.000164    192.168.99.5          1.1.1.2       TCP      0x58bb (22715) 57134      443        1683802427 0          536292142  65536      57134 > 443 [ACK] Seq=1683802427 Ack=536292142 Win=65536 [TCP CHECKSUM INCORRECT] Len=0

   2089 15:13:43.491000 463.136523  0.011644    192.168.99.5          1.1.1.2       TLSv1    0x58bc (22716) 57134      443        1683802427 119        536292142  65536      Client Hello

   2092 15:13:43.530437 463.175960  0.039437    1.1.1.2       192.168.99.5          TLSv1    0x61f5 (25077) 443        57134      536292142  853        1683802546 65536      Server Hello, Certificate, Server Key Exchange, Server Hello Done

   2093 15:13:43.569422 463.214945  0.038985    192.168.99.5          1.1.1.2       TLSv1    0x58be (22718) 57134      443        1683802546 134        536292995  64768      Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message

                Certificate (id-at-commonName=Default Web Site)  => Returned certificate is not the one expected by the client

 

   2094 15:13:43.608227 463.253750  0.038805    1.1.1.2       192.168.99.5          TLSv1    0x61f6 (25078) 443        57134      536292995  59         1683802680 65536      Change Cipher Spec, Encrypted Handshake Message

   2095 15:13:43.608869 463.254392  0.000642    192.168.99.5          1.1.1.2       TCP      0x58bf (22719) 57134      443        1683802680 0          536293054  64768      57134 > 443 [FIN, ACK] Seq=1683802680 Ack=536293054 Win=64768 [TCP CHECKSUM INCORRECT] Len=0

   2096 15:13:43.645152 463.290675  0.036283    1.1.1.2       192.168.99.5          TCP      0x61f7 (25079) 443        57134      536293054  0          1683802681 0          443 > 57134 [RST, ACK] Seq=536293054 Ack=1683802681 Win=0 Len=0

 

 

=> On the other hand when checking the certificate bindings on the UAG server side, I saw that UAG DA server IPHTTPS server was bound to 1.1.1.1:443 with the appropriate certificate

 

c:\> netsh http show sslcert

 

...

    IP:port                 : 1.1.1.1:443

    Certificate Hash        : 8691087835614a5f09b5f11c403d595c461ff603

    Application ID          : {5d8e2743-ef20-4d38-8751-7e400f200e65}

    Certificate Store Name  : MY

    Verify Client Certificate Revocation    : Enabled

    Verify Revocation Using Cached Client Certificate Only    : Disabled

    Usage Check    : Enabled

    Revocation Freshness Time : 0

    URL Retrieval Timeout   : 0

    Ctl Identifier          : 

    Ctl Store Name          : 

    DS Mapper Usage    : Enabled

    Negotiate Client Certificate    : Disabled

...

 

 

c:\> certutil -store -v my

 

...

================ Certificate 2 ================

Serial Number: 5485200900020000015c

Issuer: CN=CA-name, DC=company, DC=com

NotBefore: 18/04/2012 16:49

NotAfter: 18/04/2014 16:49

Subject: CN=UAGDAServer-FQDN

Non-root Certificate

Template: WebServer3, Web Server 3

Cert Hash(sha1): 86 91 08 78 35 61 4a 5f 09 b5 f1 1c 40 3d 59 5c 46 1f f6 03

  Key Container = d53d6468acfdb333ca5698ae67f3549e_6dd37665-5fa6-46ec-89de-8857879f2500

  Simple container name: le-WebServer3-afc38199-0bb5-4cb9-b043-50851171183d

  Provider = Microsoft Base Cryptographic Provider v1.0

Encryption test passed

...

 

The certificate seems to be the correct one but since UAGDAServer-FQDN is not mapped to 1.1.1.1 (it’s mapped to 1.1.1.2), another certificate is returned to the client in the TLS sessions and IPHTTPS connection fails as well. That was also evident in the network trace:

 

2084        15:13:43.441664     463.087187             0.002359 192.168.99.5           192.168.99.100       DNS         Standard query A UAGDAServer-FQDN

2085        15:13:43.442251     463.087774             0.000587 192.168.99.100       192.168.99.5           DNS         Standard query response A 1.1.1.2

 

I informed my customer about the problem and they changed the DNS settings. But afterwards my customer informed me that IPHTTPS connectivity problem was still in place but this time with a different error number. We re-run the previous action plan and collected logs one more time:

 

ANALYSIS 2:

=========

This time the problem was a bit different. As per the log outputs from the client and UAG DA server, there was a packet drop issue between the client and server, client’s IPHTTPS traffic wasn’t arriving at UAG DA server so the IPHTTPS tunnel establishment was failing:

 

c:\> netsh interface httpstunnel show interface

 

Interface IPHTTPSInterface (Group Policy)  Parameters

------------------------------------------------------------

Role                       : client

URL                        : https://UAGDAServer-FQDN:443/IPHTTPS

Last Error Code            : 0x274c

Interface Status           : failed to connect to the IPHTTPS server. Waiting to reconnect

 

 

 

err 0x274c

# for hex 0x274c / decimal 10060 :

  WSAETIMEDOUT                                                  winerror.h

# A connection attempt failed because the connected party did

# not properly respond after a period of time, or established

# connection failed because connected host has failed to

# respond.

  WSAETIMEDOUT                                                  winsock2.h

# 2 matches found for "0x274c"

 

 

=> And we can really see that the DA client could not connect to TCP 443:

 

No.     Time            Time since  Delta       Source                Destination           Protocol Info

   2657 14:43:33.170638 199.176755  197.498302  192.168.99.7          1.1.1.1       TCP      54007 > 443 [SYN] Seq=2446300399 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1

   2693 14:43:33.563827 199.569944  0.393189    192.168.99.7          1.1.1.1       TCP      54008 > 443 [SYN] Seq=4024014335 Win=8192 Len=0 MSS=1460 SACK_PERM=1

   2736 14:43:36.583165 202.589282  0.258635    192.168.99.7          1.1.1.1       TCP      54008 > 443 [SYN] Seq=4024014335 Win=8192 Len=0 MSS=1460 SACK_PERM=1

   2839 14:43:42.577128 208.583245  0.259486    192.168.99.7          1.1.1.1       TCP      54008 > 443 [SYN] Seq=4024014335 Win=8192 Len=0 MSS=1460 SACK_PERM=1

   3482 14:44:24.615646 250.621763  18.289976   192.168.99.7          1.1.1.1       TCP      54023 > 443 [SYN] Seq=3215921282 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1

   3516 14:44:27.621705 253.627822  3.006059    192.168.99.7          1.1.1.1       TCP      54023 > 443 [SYN] Seq=3215921282 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1

   3585 14:44:33.627033 259.633150  6.005328    192.168.99.7          1.1.1.1       TCP      54023 > 443 [SYN] Seq=3215921282 Win=8192 Len=0 MSS=1460 SACK_PERM=1

   4432 14:45:36.642904 322.649021  4.472205    192.168.99.7          1.1.1.1       TCP      54035 > 443 [SYN] Seq=55761659 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1

   4472 14:45:39.638130 325.644247  2.995226    192.168.99.7          1.1.1.1       TCP      54035 > 443 [SYN] Seq=55761659 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1

   4539 14:45:45.653327 331.659444  6.015197    192.168.99.7          1.1.1.1       TCP      54035 > 443 [SYN] Seq=55761659 Win=8192 Len=0 MSS=1460 SACK_PERM=1

   6202 14:48:00.662720 466.668837  41.395992   192.168.99.7          1.1.1.1       TCP      54064 > 443 [SYN] Seq=2998968172 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1

   6239 14:48:03.672009 469.678126  3.009289    192.168.99.7          1.1.1.1       TCP      54064 > 443 [SYN] Seq=2998968172 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1

   6309 14:48:09.673882 475.679999  6.001873    192.168.99.7          1.1.1.1       TCP      54064 > 443 [SYN] Seq=2998968172 Win=8192 Len=0 MSS=1460 SACK_PERM=1

 

=> Also none of the above requests seen on the client side trace were present on the UAG DA server side trace which meant that UAG DA server never saw them

 

My customer further checked the connectivity with their service provider and it was found out that due to an incorrect routing configuration traffic sent to 1.1.1.1:443 was really dropped/routed somewhere else and hence the IPHTTPS tunnel wasn’t established. After fixing the issue with the router configuration, the problem was resolved and IPHTTPS tunnel was successfully established.

 

Hope this helps

 

Thanks,

Murat