Introduction
This post is about two scenarios which involved ISA server and third party content application. Let’s go over each one of those and the troubleshooting steps involved in order to fix it.
Scenario 1
ISA was doing a Proxy Chain with a third party product, when the client (using Windows XP SP2 with Internet Explorer 6) was browsing a HTTPS web site behind the ISA Server the Internet Explorer crashed with an access violation. If the clients bypassed the ISA Server and use only the upstream third party proxy the issue did not happen.
Scenario 2
Clients using Windows XP SP2 were unable to update via Windows update. When we took one of the laptops and bypassed ISA which was the proxy in the corporate LAN, they were able to update without any problem. Moreover, the automatic update service (which runs in the context of SVCHOST) was crashing when we tried from LAN. ISA was forwarding web traffic to a third party product.
Troubleshooting
The data that was collected to troubleshoot these included
1) Network Capture on ISA on all adapters concerned. 2) ADPLUS crash dump for the respective processes on the client. 3) Application logs from the client machine s.
Data Analysis
For the first scenario the ISA Logging did not show up any suspicious error and the network traces pretty much shows the normal traffic:
1. The client requesting the SSL object from a Web server. 2. The “CONNECT Server_name:443 HTTP/1.1” being sent to port 8080 on the ISA Server computer. 3. The ISA Server connecting to the destination Web server on port 443. 4. The TCP connection established and the ISA Server returning the HTTP/1.1 200 connection established.
Fortunately Internet Explorer was raising an exception (c0000005 – Access Violation) and it was possible to get the dump during the crash. The exception was happening on the module wininet.dll, handling the function HTTP_REQUEST_HANDLE_OBJECT OpenProxyTunnel_Fsm on the client side
For the second scenario the error in the Application log of the client machine when automatic update failed was
Event Type: Error Event Source: Application Error Event Category: (100) Event ID: 1000 Date: 7/12/2010 Time: 2:18:22 PM User: N/A Computer: XXXX Description: Faulting application svchost.exe, version 5.2.3790.3959, faulting module winhttp.dll, version 5.2.3790.4427, fault address 0x000000000003e521.
Upon analysis of the crash dump from client machine we found:
The stored exception information can be accessed via .ecxr. (97c.ba8): Access violation - code c0000005 (first/second chance not available)
We crashed in:
4d51b7d3 c7402400000100 mov dword ptr [eax+24h],10000h ds:0023:00000024=????????
The module and function in concern was
015ef4d8 4d51b9bc winhttp!HTTP_REQUEST_HANDLE_OBJECT::OpenProxyTunnel_Fsm+0x44b
Loaded symbol image file: winhttp.dll
Image path: C:\WINDOWS\system32\winhttp.dll Image name: winhttp.dll Timestamp: Wed Aug 04 13:27:07 2004 (411096D3) CheckSum: 00055D26 ImageSize: 00058000 File version: 5.1.2600.2180 Product version: 5.1.2600.2180 File flags: 0 (Mask 3F) File OS: 40004 NT Win32 File type: 2.0 Dll File date: 00000000.00000000 Translations: 0409.04b0 CompanyName: Microsoft Corporation ProductName: Microsoft® Windows® Operating System InternalName: winhttp.dll OriginalFilename: winhttp.dll ProductVersion: 5.1.2600.2180 FileVersion: 5.1.2600.2180 (xpsp_sp2_rtm.040803-2158) FileDescription: Windows HTTP Services LegalCopyright: © Microsoft Corporation. All rights reserved.
The reason it was crashing was because the TCP connection was getting closed. Reviewing the network capture we could see the following:
Client to ISA
CONNECT www.update.microsoft.com:443 HTTP/1.0 User-Agent: Windows-Update-Agent Host: www.update.microsoft.com Content-Length: 0 Proxy-Connection: Keep-Alive
ISA to Client
HTTP/1.1 200 Connection Established Content-Length: 0 Proxy-Agent: ThirdPartyProduct/3.0 Via: 1.1 ISA Connection: Keep-Alive Proxy-Connection: Keep-Alive
The client was resetting the packet as soon as it got this packet.
Conclusion
Both the scenarios revolved around the same problem:
Proxy Connect Method is used by the client to tunnel the TCP based traffic from the proxy server. After the connection is established the proxy server simply tunnels the traffic between the client and the internet web server. To tunnel SSL traffic though proxy, browsers or any other application can make a CONNECT request to the proxy server.
More info see http://tools.ietf.org/html/draft-luotonen-web-proxy-tunneling-01#section-3.2.1
Understanding the CONNECT Method
The SSL session is established between the client generating the request, and the destination (secure) Web server; the proxy server in between is merely tunneling the encrypted data, and does not take any other part in the secure transaction.
Request
The client connects to the proxy, and uses the CONNECT method to specify the hostname and the port number to connect to. The hostname and port number are separated by a colon, and both of them must be specified.
Example:
CONNECT mail.microsoft.com:443 HTTP/1.0 User-Agent Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; MS-AOC 4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; MS-RTC EA 2; MS-RTC LM 8; .NET4.0C; .NET4.0E)
Proxy Response
The client will wait for a response from the proxy. The proxy will evaluate the request, make sure that it is valid, and that the user is authorized to request such a connection. If everything is in order, the proxy will make a connection to the destination server, and, if successful, send a "200 Connection established" response to the client
An example of a tunneling request/response in an interleaved:
CLIENT -> SERVER
SERVER -> CLIENT
CONNECT home.netscape.com:443 HTTP/1.0 User-agent: Mozilla/4.0
<<< empty line >>>
HTTP/1.0 200 Connection established Proxy-agent: Netscape-Proxy/1.1 <<< empty line >>>
<<< Data tunneling to both directions begins >>>
Although there is no specification details about Content length header in Proxy Connect Request and Response headers, never the less having a content length in the CONNECT header is not of much use. Connect is used only make an initial connection to the proxy server for requesting to tunnel traffic
Author Saurav Datta Security Support Escalation Engineer Microsoft CSS Forefront Security Edge Team
Technical Reviewer Yuri Diogenes Sr Security Support Escalation Engineer Microsoft CSS Forefront Security Edge Team
Good work Saurav..
There are many known errors with svchost.exe, below we list just a few:
SVChost.exe running multiple times
SVChost.exe using 50-100% CPU
Over 100mb ram in use by SVChost.exe
A fake SVChost.exe virus file running
The SVCHost.exe application errorif you need more detailed steps.
Try checking this guide on <a href="windowsfix.org/.../a>
Microsoft Forefront Client Crashing Internet Explorer!!!! We had to revert back to internet Explorer 6