Introduction
It could be just another case where we grab some data, look at it and resolve the issue, but it was a long road until we get to the bottom of this. The scenario was quiet simple: clients that were behind TMG couldn’t access HTTPS sites. All other sites using HTTP were working just fine and the most phantasmagoric thing that we noticed was that on TMG Logging we just have entries for 8080, no HTTPS request at all as show below:
On the client, the only thing that was showing up was:
Troubleshooting
All the usual suspects during a standard troubleshooting phase were eliminated (HTTPS Inspection, Malware Inspection, NIS, URL Filtering, HTTP Filter, Network, etc), it was time to thing: what else is left? What about the client? Let’s take a netmon on the client and see what’s going, right? Yep…let’s do it. For my surprise, after starting netmon and repro the issue, we did not even saw the HTTPS Request leaving the workstation; we just saw requests on port 8080 and then the clients sent a FIN:
Digging Deeper
In order to investigate further what was going on I used the tool TCPView, which gives more details information about network connectivity and process that are using for a specific connection. So I tried to access the Bank of America site (which uses HTTPS by default) and here it is the result:
Notice two things:
Note1: I’m not saying the driver name because this behavior was observed in many others third party AV filter drivers.
Ok, now we have some footprints that we can use to investigate further. The process XXX.exe belongs to a third party antivirus. As a temporary test we disabled the AV solution using MSConfig and restarted the workstation. It didn’t help since there were some other kernel drivers loaded in the system. As a temporary measure we uninstalled the AV, tested and worked. At that point we handled over the issue to the AV Vendor.
Note2: Only disable AV solution on the workstation for troubleshooting purpose, validate your tests and enabled it back. It is strongly recommended to have antvirus solution on your system and address potential issues with the AV vendor.
Just another reminder that not even everything that you think is caused by TMG, it actually is….this is just another example of TMG (as it was on ISA) be a victim of another problem.
Wow, 5 months already passed since the TMG book was released, looks like it was yesterday that me Mohit and Jim started this project back in March 2008. Well, time past really fast….anyway, this is a quick post to talk about our Forefront TMG Book and the great feedback that we are receiving so far from TMG Book readers around the world (more than two thousand already have this book). Besides the fact that we received very good feedbacks directly from readers, there are also some good feedbacks on sites such as Amazon and also great reviews from some folks at Microsoft, such as Alan and independent reviewers such as New Signature Creative Technology and in particular Left Brain bookstore that says:
The book is available at:
Last but not least I would like to say thank you for all of you that acquire this book and support this product.
Were you at TechEd US 2010? If you were I hope that you saw the presentation below. If you were not, then here it is a good chance to watch it and learn more about TMG.
PPT Slides are located at http://ecn.channel9.msdn.com/o9/te/NorthAmerica/2010/pptx/SIA308.pptx
Video is available for download at http://ecn.channel9.msdn.com/o9/te/NorthAmerica/2010/wmv/SIA308.wmv
Here it is another set of slides, now about TMG from a presentation that I delivered last year in Brazil.
Once again, enjoy it.
Today a friend of mine asked where he can get the slides that once were available at http://www.microsoft.com/brasil/technet/eventos/ciclocontinuado/ciclo-nlb.mspx, the slides are not there anymore, so I decided to share it here:
Enjoy it.
We just released a new KB where we explain more details about Malware Inspection and NIS files. Check it out at http://support.microsoft.com/kb/2160835.
Today I was assisting a friend of mine here from TMG team that was facing this issue, same issue that was also mentioned on this thread. The problem was happening when using Cryptography Next Generation (CNG) or also called V3, TMG was not recognizing the private key and was showing up this error message. This is a known issue because TMG (and ISA) don’t support CNG (V3 Certificates). This is well documented under the unsupported documentation here:
Issue: Forefront TMG does not support the use of certificates created using CNG (Certificate New Generation) based templates for Web listeners or as client certificate authentication in Web publishing or Web chaining rules.
Cause: CNG certificates are not usable by Forefront TMG.
Workaround: Create certificates using Windows 2000 or Windows 2003 templates.
From: http://technet.microsoft.com/en-us/library/ee796231.aspx#dfg9o9i8uuy6tre
Again, make sure to read this unsupported document before deploy TMG, there you will find the official statement from TMG Product Team about what it is supposed to work and what it is not.
Note: Important to emphasize that CNG V3 is not X.509 V3. CNG V3 refers to the new V3 Certificate Template on 2008 while X.509 V3 is the current certificate standard in which TMG is fully compatible.
Recently I was engaged on this TechNet Forum thread and I thought that it was an interesting question and concern. Let me explain the scenario on which I was able to repro the behavior on my own lab:
Contoso has a security policy where only users that belong to the Internet Users group can access Internet. To enforce that they created a rule on ISA Server on which only members of the Internet Group can access Internet. Contoso uses Internet Explorer 6 on all workstations of their network. The process today is that once the user receives authorization to access Internet, the AD Admin adds the user to Internet Users group and the user just need to open a new browser session and he will have access. Recently Contoso upgraded 50% of their workstations to use Internet Explorer 8, they notice the following behavior since this change: if an user that doesn’t belong to Internet Users group is already logged on the workstation and the administrator add this user to the Internet User’s group, the user receives an error when try to browse Internet, the error says that ISA denied the request. If the user logs off from the windows and logon again it works fine.
Why this is happening?
What it is happening on this case is simply the way that Internet Explorer 8 works for the authentication part of the conversation. Since the version 7, Internet Explorer was capable to use Kerberos for proxy authentication (while IE6 and lower only use NTLM and Basic). ISA Server will negotiate the authentication with the browser and will authenticate according to the method that is supported on both sides. The difference in this case is the following:
With IE7 (and higher)
1. Client sends the GET request to www.microsoft.com (for example). This request goes as anonymous.
2. ISA will send the 407 asking for authentication.
3. If client has already a Kerberos cached ticket it will not go to the DC to get a new one, if there is no ticket then it will go to the DC to get an updated ticket to send to ISA.
4. Client will send another GET Request now with the credentials and the ticket.
5. ISA will verify the request and allow (or deny according to the rule).
Note: in this case since ISA doesn't go to the DC, it relies on user's token to access the resource. Since the current ticket doesn’t include the update on user’s group membership, ISA will verify that the user doesn't below to the group and will deny the request.
With IE6
3. Client sends another GET request with the credentials (NTLM).
4. ISA goes to the DC to authenticate the user.
Note: since ISA goes to the DC it will get an updated version of the user's group membership and will verify that the user now belongs to this group.
5. ISA will allow the user to pass through.
Under the Hood
Prior to get to this conclusion I decided to do some tests to validate the theory and the tests worked pretty fine. Here it is the traffic from the client workstation with the user “Yuri” logged in, right after the inclusion of the user to Internet User’s group:
1. Client sends the GET Request
10.20.20.201 10.20.20.1 HTTP HTTP:Request, GET http://www.microsoft.com/
2. ISA asks for authentication:
10.20.20.1 10.20.20.201 HTTP HTTP:Response, HTTP/1.1, Status Code = 407, URL: http://www.microsoft.com/
Connection: Authentication
ProtocolVersion: HTTP/1.1
StatusCode: 407, Proxy authentication required
Reason: Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy filter is denied. )
Via: 1.1 ISACONTN1
+ ProxyAuthenticate: Negotiate
Proxy-Authenticate:
+ ProxyAuthenticate: Kerberos
+ ProxyAuthenticate: NTLM
Connection:
Connection: Keep-Alive
ProxyConnection: Keep-Alive
Pragma: no-cache
Cache-Control: no-cache
+ ContentType: text/html
ContentLength: 4113
HeaderEnd: CRLF
+ payload: HttpContentType = text/html
Note: Notice that ISA is negotiating the authentication method.
3. Client sends another GET request now with the credential.
10.20.20.201 10.20.20.1 HTTP HTTP:Request, GET http://www.microsoft.com/ , Using SPNEGO Authorization
Command: GET
+ URI: http://www.microsoft.com/
Accept: */*
Accept-Language: en-us
UserAgent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322)
Accept-Encoding: gzip, deflate
Cookie: MUID=42623805587D4F6EB894604864D99E47; WT_FPC=id=173.74.31.197-401582496.30090674:lv=1279361835221:ss=1279361835221; MC1=GUID=ba325a1581e1aa4dbcd0988320053feb&HASH=155a&LV=20107&V=3; A=I&I=AxUFAAAAAABDBgAAgb+LoGxG8UwjOO2L0QId7Q!!
- ProxyAuthorization: Negotiate
- Authorization: Negotiate YIIE5QYGKwYBBQUCoIIE2TCCBNWgJDAiBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKwYBBAGCNwICCqKCBKsEggSnYIIEowYJKoZIhvcSAQICAQBuggSSMIIEjqADAgEFoQMCAQ6iBwMFACAAAACjggO7YYIDtzCCA7OgAwIBBaEOGwxDT05UT1NPLk1TRlSiKTAnoAMCAQKhIDAeGwRIVFRQGxZpc2Fjb250bjEuY
WhiteSpace:
- NegotiateAuthorization:
Scheme: Negotiate
- GssapiKrb5: 0x1
Kerberos:
Note: Notice that client is using Kerberos.
4. This ticket doesn’t contain an updated information about user’s group membership, hence ISA Server sends a deny.
10.20.20.1 10.20.20.201 HTTP HTTP:Response, HTTP/1.1, Status Code = 502, URL: http://www.microsoft.com/
StatusCode: 502, Bad gateway
Reason: Proxy Error ( The ISA Server denied the specified Uniform Resource Locator (URL). )
Connection: close
ProxyConnection: close
ContentLength: 4059
5. At this point, I ran the command klist on the workstation and here are the tickets that I had at that point:
C:\Program Files\Windows Resource Kits\Tools>klist tickets
Cached Tickets: (3)
Server: krbtgt/CONTOSO.MSFT@CONTOSO.MSFT
KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)
End Time: 7/17/2010 18:34:46
Renew Time: 7/24/2010 8:34:46
Server: HTTP/isacontn1.contoso.msft@CONTOSO.MSFT
Server: host/client1.contoso.msft@CONTOSO.MSFT
6. I thought to myself: well, if I wipe all the tickets I will force the user to go to the DC and get an updated ticket, this way if I try again to access Internet, I will use the updated ticket. To do that I ran the command: klist purge. I deleted all tickets and the end result was:
C:\Program Files\Windows Resource Kits\Tools>klist purge
Cached Tickets: (0)
Now is time for the truth, tried to access Internet again and then it WORKED. Here it is the netmon trace for this second attempt with my Kerberos ticket cache clean:
3. Client goes to the DC to get ticket for the user “Yuri”:
10.20.20.201 10.20.20.20 KerberosV5 KerberosV5:AS Request Cname: yuri Realm: CONTOSO.MSFT Sname: krbtgt/CONTOSO.MSFT
- Kerberos: AS Request Cname: yuri Realm: CONTOSO.MSFT Sname: krbtgt/CONTOSO.MSFT
- AsReq: Kerberos AS Request
+ ApplicationTag:
+ KdcReq: KRB_AS_REQ (10)
4. The DC responds:
10.20.20.20 10.20.20.201 KerberosV5 KerberosV5:AS Response Ticket[Realm: CONTOSO.MSFT, Sname: krbtgt/CONTOSO.MSFT]
- Kerberos: AS Response Ticket[Realm: CONTOSO.MSFT, Sname: krbtgt/CONTOSO.MSFT]
- AsRep: Kerberos AS Response
+ KdcRep: KRB_AS_REP (11)
5. Client goes again to the DC to get a ticket to the HTTP Service (ISA SPN):
10.20.20.201 10.20.20.20 KerberosV5 KerberosV5:TGS Request Realm: CONTOSO.MSFT Sname: HTTP/isacontn1.contoso.msft
- Kerberos: TGS Request Realm: CONTOSO.MSFT Sname: HTTP/isacontn1.contoso.msft
- TgsReq: Kerberos TGS Request
+ KdcReq: KRB_TGS_REQ (12)
6. The DC responds:
10.20.20.20 10.20.20.201 KerberosV5 KerberosV5:TGS Response Cname: Yuri
- Kerberos: TGS Response Cname: Yuri
- TgsRep: Kerberos TGS Response
- KdcRep: KRB_TGS_REP (13)
+ SequenceHeader:
+ Tag0:
+ PvNo: 5
+ Tag1:
+ MsgType: KRB_TGS_REP (13)
+ Tag3:
+ Crealm: CONTOSO.MSFT
+ Tag4:
+ Cname: Yuri
+ Tag5:
+ Ticket: Realm: CONTOSO.MSFT, Sname: HTTP/isacontn1.contoso.msft
+ Tag6:
+ EncPart:
7. Client sends the GET request to ISA now with the new ticket:
Host: www.microsoft.com
- Authorization: Negotiate YIIFDQYGKwYBBQUCoIIFATCCBP2gJDAiBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKwYBBAGCNwICCqKCBNMEggTPYIIEywYJKoZIhvcSAQICAQBuggS6MIIEtqADAgEFoQMCAQ6iBwMFACAAAACjggPjYYID3zCCA9ugAwIBBaEOGwxDT05UT1NPLk1TRlSiKTAnoAMCAQKhIDAeGwRIVFRQGxZpc2Fjb250bjEuY
8. ISA allows the traffic:
10.20.20.1 10.20.20.201 HTTP HTTP:Response, HTTP/1.1, Status Code = 302, URL: http://www.microsoft.com/
Conclusion
As you could see on this post this is not really an issue, neither on ISA nor on Internet Explorer, this is actually a behavioral change that Internet Explorer 7 and higher had. This is a good improvement, because Kerberos is much lighter and doesn’t put too much pressure on the DC. One classic problem with NTLM is that the authentication request against the DC is huge, read the article Improving Web Proxy Client Authentication Performance on ISA Server 2006 that I wrote couple of years ago and you will see the advantage that IE7 has over IE6.
But, if it is a requirement for your business that changes to group membership immediately reflect on user’s Internet browsing experience than you can use ISA’s IP address on the IE setting rather than the name, this way you are forcing to use NTLM rather than Kerberos. Just keep in mind that when too much pressure is added on the DC you can face the infamous “random prompt for authentication” on ISA due the 5783 and 5719 events.
Vacation is always good, isn’t it? I was on vacation in June and couldn’t write too much but now I’m back and fully charged for another TMG semester. Last week I was delivering a TMG workshop for Microsoft Premier Customers and some interesting questions appeared during those three days. Among all those questions there were two that I would like to share it here:
1) Can I use ISP-Redundancy to send HTTP through one link and SMTP through another?
No, you can’t. This is unsupported and documented here.
2) Can I install TMG Firewall on EMS?
No, you can’t. This is also part of the unsupported document that you can find here.
It is very important to read this unsupported document before make decisions on your environment. Matter of fact one of the slides that I presented during this workshop last week says:
If you want to be on the safe side never skip phases and during the planning phase make sure to read the unsupported document to see if there is any scenario that might affect your supportability state.
Have a great TMG deployment.
As we just announced on Microsoft Press blog, three new Forefront eBooks are coming soon, check it out the full blog MS Press Blog. Here are they:
We hope you enjoy it.
Cheers !!