Consider a scenario where you are publishing a third party web server, in this case an Apache Server that uses HTTPS through ISA Server 2006. Randomly the site doesn’t work, clients are unable to access it and when this happens the publishing rule test button shows the result below (error 0x80090326):
Notice that this error talks about a server certificate error, so clearly it is something during the SSL process. Reviewing the Data Using ISA Data Packager in repro mode (web proxy / web publishing template) was possible to collect simultaneous netmon traces from both NICs (internal and external). During those traces it was possible to see that the SSL handshake on the external interface using the certificate that was bound to the Web Listener on ISA was working just fine. Reviewing the SSL Handshake on the internal interface, while ISA was negotiating with the published server (Apache) we had a failure. Here it is the moment of the failure: ISA APACHE TCP TCP:Flags=......S., SrcPort=24433, DstPort=443, PayloadLen=0, Seq=3108278462, Ack=0, Win=65535 ( ) = 65535 APACHE ISA TCP TCP:Flags=...A..S., SrcPort=443, DstPort=24433, PayloadLen=0, Seq=2120534540, Ack=3108278463, Win=5840 ( Scale factor not supported ) = 5840 ISA APACHE TCP TCP: Flags=...A...., SrcPort=24433, DstPort=443, PayloadLen=0, Seq=3108278463, Ack=2120534541, Win=65535 (scale factor 0x0) = 65535 After finishing the TCP Handshake they start the SSL handshake and this is done by ISA sending the SSL Client Hello as shown below: ISA APACHE TLS TLS:TLS Rec Layer-1 HandShake TLSSSLData: Transport Layer Security (TLS) Payload Data - TLS: TLS Rec Layer-1 HandShake - TlsRecordLayer: TLS Rec Layer-1 HandShake ContentType: HandShake - Version: TLS 1.0 Major: 3 (0x3) Minor: 1 (0x1) Length: 88 (0x58) - SSLHandshake: SSL HandShake ClientHello(0x01) HandShakeType: ClientHello(0x01) Length: 84 (0x54) - ClientHello: TLS 1.0 + Version: TLS 1.0 + RandomBytes: SessionIDLength: 16 (0x10) SessionID: Binary Large Object (16 Bytes) CipherSuitesLength: 22 + TLSCipherSuites: TLS_RSA_WITH_RC4_128_MD5 { 0x00,0x04 } + TLSCipherSuites: TLS_RSA_WITH_RC4_128_SHA { 0x00,0x05 } + TLSCipherSuites: TLS_RSA_WITH_3DES_EDE_CBC_SHA { 0x00,0x0A } + TLSCipherSuites: TLS_RSA_WITH_DES_CBC_SHA { 0x00,0x09 } + TLSCipherSuites: TLS_NTRU_NSS_WITH_AES_256_CBC_SHA { 0x00, 0x64 } + TLSCipherSuites: TLS_NTRU_NSS_WITH_3DES_EDE_CBC_SHA { 0x00, 0x62 } + TLSCipherSuites: TLS_RSA_EXPORT_WITH_RC4_40_MD5 { 0x00,0x03 } + TLSCipherSuites: TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5 { 0x00,0x06 } + TLSCipherSuites: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA { 0x00,0x13 } + TLSCipherSuites: TLS_DHE_DSS_WITH_DES_CBC_SHA { 0x00,0x12 } + TLSCipherSuites: TLS_NTRU_NSS_WITH_AES_128_CBC_SHA { 0x00, 0x63 } CompressionMethodsLength: 1 (0x1) CompressionMethods: 0 (0x0) ExtensionsLength: 5 (0x5) Right after that Apache sends a SSL Encrypt Alert error as shown below: APACHE ISA TLS TLS:TLS Rec Layer-1 Encrypted Alert TLSSSLData: Transport Layer Security (TLS) Payload Data - TLS: TLS Rec Layer-1 Encrypted Alert - TlsRecordLayer: TLS Rec Layer-1 Encrypted Alert ContentType: Encrypted Alert - Version: TLS 1.0 Major: 3 (0x3) Minor: 1 (0x1) Length: 2 (0x2) EncryptedData: Binary Large Object (2 Bytes) 15 03 01 00 02 02 2F ....../ By looking to the last two bytes of the hex value under Encrypted data we can find the meaning of the alert: o 2F in Hex = 47 in Decimal o 47 maps to illegal_parameter(47) error according to TLS RFC (http://www.ietf.org/rfc/rfc2246.txt?number=2246) Note: thanks to sudeepg for this nice approach reading SSL Encrypt Alert. Apache Server is saying that the TLS SSL Client Hello sent by ISA as an illegal parameter for this SSL negotiation. Conclusion This problem can happen because MS10-049 which is installed on ISA Server. As a temp workaround this update was removed and the issue got resolved on this particular scenario. However, ultimately you should not remove this update; you should talk to the third party company web server admin and discuss the CVE-2009-3555 with him and how their product adequate for that. If you are publishing an IIS Server you might have this issue too, if you do, read the post below to see how to fix it: http://blogs.msdn.com/b/jpsanders/archive/2010/09/08/understanding-problems-with-ms10-049-kb-980436-and-ietf-rfc5746.aspx
Notice that this error talks about a server certificate error, so clearly it is something during the SSL process.
Reviewing the Data
Using ISA Data Packager in repro mode (web proxy / web publishing template) was possible to collect simultaneous netmon traces from both NICs (internal and external). During those traces it was possible to see that the SSL handshake on the external interface using the certificate that was bound to the Web Listener on ISA was working just fine. Reviewing the SSL Handshake on the internal interface, while ISA was negotiating with the published server (Apache) we had a failure. Here it is the moment of the failure:
ISA APACHE TCP TCP:Flags=......S., SrcPort=24433, DstPort=443, PayloadLen=0, Seq=3108278462, Ack=0, Win=65535 ( ) = 65535
APACHE ISA TCP TCP:Flags=...A..S., SrcPort=443, DstPort=24433, PayloadLen=0, Seq=2120534540, Ack=3108278463, Win=5840 ( Scale factor not supported ) = 5840
ISA APACHE TCP TCP: Flags=...A...., SrcPort=24433, DstPort=443, PayloadLen=0, Seq=3108278463, Ack=2120534541, Win=65535 (scale factor 0x0) = 65535
After finishing the TCP Handshake they start the SSL handshake and this is done by ISA sending the SSL Client Hello as shown below:
ISA APACHE TLS TLS:TLS Rec Layer-1 HandShake
TLSSSLData: Transport Layer Security (TLS) Payload Data
- TLS: TLS Rec Layer-1 HandShake
- TlsRecordLayer: TLS Rec Layer-1 HandShake
ContentType: HandShake
- Version: TLS 1.0
Major: 3 (0x3)
Minor: 1 (0x1)
Length: 88 (0x58)
- SSLHandshake: SSL HandShake ClientHello(0x01)
HandShakeType: ClientHello(0x01)
Length: 84 (0x54)
- ClientHello: TLS 1.0
+ Version: TLS 1.0
+ RandomBytes:
SessionIDLength: 16 (0x10)
SessionID: Binary Large Object (16 Bytes)
CipherSuitesLength: 22
+ TLSCipherSuites: TLS_RSA_WITH_RC4_128_MD5 { 0x00,0x04 }
+ TLSCipherSuites: TLS_RSA_WITH_RC4_128_SHA { 0x00,0x05 }
+ TLSCipherSuites: TLS_RSA_WITH_3DES_EDE_CBC_SHA { 0x00,0x0A }
+ TLSCipherSuites: TLS_RSA_WITH_DES_CBC_SHA { 0x00,0x09 }
+ TLSCipherSuites: TLS_NTRU_NSS_WITH_AES_256_CBC_SHA { 0x00, 0x64 }
+ TLSCipherSuites: TLS_NTRU_NSS_WITH_3DES_EDE_CBC_SHA { 0x00, 0x62 }
+ TLSCipherSuites: TLS_RSA_EXPORT_WITH_RC4_40_MD5 { 0x00,0x03 }
+ TLSCipherSuites: TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5 { 0x00,0x06 }
+ TLSCipherSuites: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA { 0x00,0x13 }
+ TLSCipherSuites: TLS_DHE_DSS_WITH_DES_CBC_SHA { 0x00,0x12 }
+ TLSCipherSuites: TLS_NTRU_NSS_WITH_AES_128_CBC_SHA { 0x00, 0x63 }
CompressionMethodsLength: 1 (0x1)
CompressionMethods: 0 (0x0)
ExtensionsLength: 5 (0x5)
Right after that Apache sends a SSL Encrypt Alert error as shown below:
APACHE ISA TLS TLS:TLS Rec Layer-1 Encrypted Alert
- TLS: TLS Rec Layer-1 Encrypted Alert
- TlsRecordLayer: TLS Rec Layer-1 Encrypted Alert
ContentType: Encrypted Alert
Length: 2 (0x2)
EncryptedData: Binary Large Object (2 Bytes)
15 03 01 00 02 02 2F
....../
By looking to the last two bytes of the hex value under Encrypted data we can find the meaning of the alert:
o 2F in Hex = 47 in Decimal
o 47 maps to illegal_parameter(47) error according to TLS RFC (http://www.ietf.org/rfc/rfc2246.txt?number=2246)
Note: thanks to sudeepg for this nice approach reading SSL Encrypt Alert.
Apache Server is saying that the TLS SSL Client Hello sent by ISA as an illegal parameter for this SSL negotiation.
Conclusion
This problem can happen because MS10-049 which is installed on ISA Server. As a temp workaround this update was removed and the issue got resolved on this particular scenario. However, ultimately you should not remove this update; you should talk to the third party company web server admin and discuss the CVE-2009-3555 with him and how their product adequate for that. If you are publishing an IIS Server you might have this issue too, if you do, read the post below to see how to fix it:
http://blogs.msdn.com/b/jpsanders/archive/2010/09/08/understanding-problems-with-ms10-049-kb-980436-and-ietf-rfc5746.aspx
Consider the following scenario:
In this scenario, when the Web server receives an HTTP request, it redirects the request to the TMG adding the https on the new location within the header as shown below:
- GET Request sent from TMG to the internal Server:
Http: Request, GET /default.aspx Command: GET + URI: /default.aspx ProtocolVersion: HTTP/1.1 Via: 1.1 TMG Host: contoso.com Accept: */* Accept-Language: en-us Connection: Keep-Alive Accept-Encoding: peerdist HeaderEnd: CRLF
- Web Server reply with the new location:
Http: Response, HTTP/1.1, Status: Moved temporarily, URL: /default.aspx ProtocolVersion: HTTP/1.1 StatusCode: 302, Moved temporarily Reason: Found Cache-Control: private Location: https://contoso.com/default.aspx Server: Microsoft-IIS/7.5 XAspNetVersion: 2.0.50727 XPoweredBy: ASP.NET ContentLength: 149 HeaderEnd: CRLF
Problem: TMG receives the request with the new location and instead of sending this new location to the client workstation, it sends http://contoso.com/default.aspx (removing the “s”), client receives this 302 and send the request again, causing an eternal loop.
Resolution: in order to fix this problem, use the resolution (method 2) from KB http://support.microsoft.com/kb/924373. Although the KB doesn’t have Forefront TMG 2010 listed, the same approach applies to TMG 2010 (yes, we will update the KB).
Today we are making publicly available the Software Update 1 Rollup 2 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 1. This hotfix include resolution for the following issues:
article
Title
2452980 (http://support.microsoft.com/kb/2452980)
Upload speed through Forefront TMG 2010 is very slow on a high speed Internet connection
2478286 (http://support.microsoft.com/kb/2478286)
Connection does not time out after inactivity time elapses in an OWA 2010 client connected to Exchange Server 2010 if published by using Forefront TMG 2010
2484988 (http://support.microsoft.com/kb/2484988)
A DNS server publishing rule stops working for a DNS server that is published by using Forefront TMG 2010
2478297 (http://support.microsoft.com/kb/2478297)
User Activity reports that are created by Forefront TMG 2010 show a wrong value in the reported data range
Notice that the first issue on this KB is the same that we were discussing on this TechNet thread here. So if you are facing such issue, make sure to install this update and run the script from KB2452980 (http://support.microsoft.com/kb/2452980). The other issue that we address on this rollup was raised from one of my customers as a problem, working with him I was able to reproduce the issue and after a long investigation we were able to find the root cause of the problem (in a great partnership with Exchange Team and TMG Developers), read KB2478286 (http://support.microsoft.com/kb/2478286) for more details. The third issue that we address on this TMG rollup is a DNS publication that stops to work, see KB2484988 (http://support.microsoft.com/kb/2484988) for more details. Last but not least a problem on the user activity report, simple stuff but that bothers for sure; see KB2478297 (http://support.microsoft.com/kb/2478297) for more details.
For ISA Server we are releasing the ISA Server 2006 hotfix package: December 2010, which includes the following updates:
KB article
2478307 (http://support.microsoft.com/kb/2478307)
MAPI client does not connect to an Exchange server on an internal network through an ISA Server 2006-based VPN connection on a computer that is running Windows 7
2481980 (http://support.microsoft.com/kb/2481980)
Unexpected authentication prompts while you use an OWA website that is published by using ISA Server 2006 SP1 if RSA authentication and FBA are used
Go get it and enjoy your holidays.
Merry XMas !!