Disclosure - the content for this post comes from Jason on our escalation services team. He has provided permission to post.
March 30 - Forgive the delete and repeat posting but the images were on internal links. This should be resolved now.
Some Microsoft Office Communicator (MOC) clients are not able to establish a TLS connection to the Office Communications Server (OCS). Customer has noticed that the issue seems to occur more with one business group as compared to other business groups within the organization but has not been able to determine what is unique/different with the users or computers affected.
Network traces from a working and failing computer. iDNA of Office Communicator during a failed connection attempt.
Network Trace Analysis
Network Trace filtered (tcp.port==5061)
What happens between frame 316 and frame 959? When a client is establishing a TLS session the client gets a copy of the server's certificate, validates it is issued from a trusted authority, and then the client verifies the certificate has not been revoked by downloading the Certificate Revocation List (CRL). If we looked at the certificate configured on the server, we could see the CRL was published at http://crl.verisign.com/pca3.crl.
Modifying the filter to include http traffic: tcp.port==5061 || http
Now we can see what was going on in the time between the last data sent by the server and the RESET sent by the server.
The next thing we should see is the client sending a GET with a NTLM Challenge Response. The client never sends this packet. We need to determine why the client did not reply. We don't know if the client application did not receive data captured in frame 417. We know the data came across the NIC but from that we cannot prove the application received it. So I have two theories: 1. The client never saw the data in frame 417 and that is why he never answers or 2. The client received the data in frame 417 and for some reason did not like it.
Generally speaking there are two sets of APIs used to communicate over the HTTP protocol: WinInet and WinHttp. In this case we don't know which one Office Communicator is using. One way you can guess which one a particular application is using is to list the modules loaded in the process. You can do this on a running system using "tasklist /m /fi "imagename eq communicator.exe" or using "lm" in a .run or .dump file. Since I have the iDNA (.run file) I listed the loaded modules (lm) and found the wininet.dll was loaded and winhttp was not. With this we can set a breakpoint on the wininet function that would go get content based on a URL. Quick search of the Platform SDK shows that InternetOpenURL does this. Setting a bp on this function and starting the run file (g).
First hit on breakpoint:
# ChildEBP RetAddr
00 05c2f51c 0818420f wininet!InternetOpenUrlA
WARNING: Stack unwind information not available. Following frames may be wrong.
01 05c2f55c 0818266f eecrlrev+0x420f
02 05c2f6a0 0818231d eecrlrev+0x266f
03 05c2f6b8 08181a81 eecrlrev+0x231d
04 05c2f6d8 0818131e eecrlrev+0x1a81
05 05c2f738 081846ac eecrlrev+0x131e
06 05c2f840 77a96075 eecrlrev+0x46ac
07 05c2f940 77a95f31 crypt32!VerifyDefaultRevocation+0x1d0
08 05c2f9b0 77a96b04 crypt32!CertVerifyRevocation+0xb7 [frames removed]
0:005> x lpszUrl
05c2f528 lpszUrl = 0x05c2f58c http://crl.verisign.com/pca3.crl
Based on the call stack, we can see the call to CertVerifyRevocation led to the call to InernetOpenUrl. Dumping out the URL passed to the function, we validate it was the URL for the CRL.
From here I wanted to see us put the request on the wire. Set a breakpoint on ws2_32!send in order to see the data as it left the application.
When we hit the bp we can see the following in our outbound buffer.
0:005> x buf
05c2f100 buf = 0x00155420 "GET http://crl.verisign.com/pca3.crl HTTP/1.0..User-Agent: Entrust Certificate Revocation..Host: crl.verisign.com..Pragma: no-cache..Cookie: BCSI-CS0AEE9E8D=2..Proxy-Authorization: NTLM TlRMTVNTUAABAAAAB7IIogUABQAwAAAACAAIACgAAAAFASgKAAAAD1Q2Q0s5NDNZTUFQTEU=...."
We can see this is the buffer sent in the network trace, frame 416 (Figure 2)
Setting bp on the winsock receive fuction stepping to the point that the buffer is filled out.
Buffer address was @ 00170aa8
0:005> da 00170aa8
00170aa8 "HTTP/1.1 407 Proxy Authenticatio"
00170ac8 "n Required..Proxy-Authenticate: "
00170ae8 "NTLM TlRMTVNTUAACAAAABAAEADgAAAA"
00170be8 "-Control: no-cache..Pragma: no-c"
00170c08 "ache..Content-Type: text/html; c"
This aligns to frame 417 in figure 2
So at this point we know the client application (MOC) received the 407 proxy response with the NTLM challenge. From here we should expect the client to turn around and send the NTLM challenge response. We still have a bp on send and I added a bp for closesocket to catch anything going wrong if we don't send the response.
My CloseSocket bp hit.
05c2edc0 771bfb5a ws2_32!closesocket
05c2edf8 771ce87f wininet!ICSocket::Close+0x25
05c2ee08 771ce8d2 wininet!ICSocket::Disconnect+0x23
05c2ee24 771c5061 wininet!HTTP_REQUEST_HANDLE_OBJECT::ReleaseConnection+0x6d
05c2ee48 771c8c6c wininet!HTTP_REQUEST_HANDLE_OBJECT::CloseConnection+0x84
05c2f194 771c9d37 wininet!HTTP_REQUEST_HANDLE_OBJECT::ReadData_Fsm+0x5e8
05c2f1a0 771bcb14 wininet!CFsm_ReadData::RunSM+0x2e
05c2f1b8 771bcac2 wininet!CFsm::Run+0x39
05c2f1d0 771c9cb5 wininet!DoFsm+0x25
05c2f1e0 771e4050 wininet!HTTP_REQUEST_HANDLE_OBJECT::ReadData+0x38
05c2f20c 771bcc1d wininet!HTTP_REQUEST_HANDLE_OBJECT::HttpSendRequest_Start+0x134
05c2f220 771bcb14 wininet!CFsm_HttpSendRequest::RunSM+0x59
05c2f238 771bcac2 wininet!CFsm::Run+0x39
05c2f250 771bccc5 wininet!DoFsm+0x25
05c2f278 771c60be wininet!HttpWrapSendRequest+0x140
05c2f29c 771c5fc3 wininet!HttpSendRequestA+0x1d
05c2f3f4 771bcb14 wininet!ParseHttpUrl_Fsm+0x22f
05c2f40c 771bcac2 wininet!CFsm::Run+0x39
05c2f424 771c5d2a wininet!DoFsm+0x25
05c2f494 771c5b92 wininet!ParseUrlForHttp_Fsm+0x214
05c2f4a0 771bcb14 wininet!CFsm_ParseUrlForHttp::RunSM+0x2b
05c2f4b8 771bcac2 wininet!CFsm::Run+0x39
05c2f4d0 771c5a42 wininet!DoFsm+0x25
05c2f51c 0818420f wininet!InternetOpenUrlA+0x1d7
05c2f55c 0818266f eecrlrev+0x420f
05c2f6a0 0818231d eecrlrev+0x266f
05c2f6b8 08181a81 eecrlrev+0x231d
05c2f6d8 0818131e eecrlrev+0x1a81
05c2f738 081846ac eecrlrev+0x131e
05c2f840 77a96075 eecrlrev+0x46ac
05c2f940 77a95f31 crypt32!VerifyDefaultRevocation+0x1d0
05c2f9b0 77a96b04 crypt32!CertVerifyRevocation+0xb7
We see here the client closed the socket and if we step back to the wininet!HTTP_REQUEST_HANDLE_OBJECT::CloseConnection function we can see (code review) the disconnect reason was due to the client inability to support the server mandate for Keep-Alive.
Enable HTTP 1.1 and HTTP 1.1 through proxy under Internet Options > Advanced. Using HTTP 1.1. enables the client to support the http Keep-Alive options which is required to support NTLM Proxy Authentication.
Very good troubleshooting steps!