Consider a scenario where you have a client workstation behind Forefront TMG 2010 and you are trying to download files from a FTP Server. You are successfully able to logon on the FTP but after type the command “dir” you get the error message below:
The message is pretty clear about what is going on, isn’t it? Well, it is but where do I enable this option? I don’t remember having this on ISA!! To address this issue you just need to enable a new option that we have on TMG, this option is located on the FTP Filter properties as shown below:
After enabling this option and apply the changes you should be able to list your files just fine. It is important to mention that this setting has nothing to do with the FTP Read Only option, that you had in ISA 2004/2006 and still have it on TMG. The FTP Filter when running in read only mode (see figure below) will blocks all commands in the control channel except the following ones:
“ABOR, ACCT, CDUP, CWD /0, FEAT, HELP, LANG, LIST, MODE, NLST, NOOP, PASS, PASV, PORT, PWD /0, QUIT, REIN, REST, RETR, SITE, STRU, SYST, TYPE, USER, XDUP, XCWD, XPWD, SMNT”
You can customize this list by using the sample script below (from Configuring Add-ins MSDN article),in this example the script configures FTP Access Filter to allow only the USER and PASS commands:
Note: don’t change the default Read Only commands unless you have a real business need for that.
Next February 9th 11:30 AM CST I will be delivering a presentation about Troubleshooting ISA Server 2006 Performance issues for Microsoft Partners, if you are a partner and deal with ISA Server you should watch this presentation. Here it is the agenda with the core topics that will be covered:
The registration is open at https://training.partner.microsoft.com/learning/app/management/LMS_ActDetails.aspx?UserMode=0&ActivityId=573031
See you there !!
TMG installation problems can be a bit trick to troubleshooting if you don’t know which components are involved, however if you know then things start make more sense. Most of the setup problems that I faced up to now on TMG 2010 (since RTM) were related to ADLDS or some kind of domain connectivity problem. The most two recent examples are described on two articles that I tech reviewed from my friends Bala Natarajan and Niladri Dasgupta wrote:
Last week I worked on an issue where TMG admin was not able to install this brand new TMG to be used as Edge Firewall. The error message that he was receiving was:
After this error the setup process rolled back and finish without completing the installation. As recommended on the previous two articles mentioned in this post, the first step is to review the setup logs and look for more information in order to move the troubleshooting further. In the ADAM Log file we can see the following entry:
When you see an error where trust relationship between client and domain is failing, be sure to do your homework, in other words, check:
When I hit the third test I found out the problem:
This was the problem, because Windows (where I was trying to install TMG) was sending the traffic to the wrong interface. Once we moved the Internal to the top, flushdns (with ipconfig) and ran the setup again the issue went away and the installation finished successfully.
Note: same recommendation to have Internal on the top applies to UAG, check it out a great reference on that written by Jason Jones at http://blog.msedge.org.uk/2010/04/recommended-network-card-configuration_14.html
I created this blog back in February 2008 and since that day I really tried to bring to you interesting troubleshooting techniques based on real scenarios. This blog was always something that I drove on my own free time (not that I have lot of free time), but I tried to managed my time in such way that posting here was part of my regular agenda. The numbers below show how much the traffic increased over the last couple of years in this blog and I would like to thank you all for contributing with that, is because I know you are reading that I feel energized to keep writing.
I can safely say that 90% of the posts that I wrote for this blog were related to ISA/TMG, which makes a lot of sense to me as I was working for CSS Forefront Edge Team. Yes, you read it right, I “was”. Starting Monday (Feb 14th) I will be fully dedicated to the Windows Security Team as a Technical Writer. As one of the co-authors of the Forefront TMG Administrator’s Companion Book and Forefront TMG Deployment Guide, I plan to keep writing about Forefront TMG here, but certainly will not be on the same frequency as before since I will be dedicated to Windows Security subject. From now on I will be more engaged in produce content that will be available in other locations, such as:
http://social.technet.microsoft.com/wiki/contents/articles/wiki-it-security-portal.aspx
There are some initiatives on the Forefront TMG space that I’m still engaged during this transition phase, which are:
Again, thanks a lot for visiting this blog and I hope to keep partnering with you in 2011, now in a broader way.
Stay Safe!!
Today we are making publicly available the Software Update 1 Rollup 2 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 1. This hotfix include resolution for the following issues:
article
Title
2452980 (http://support.microsoft.com/kb/2452980)
Upload speed through Forefront TMG 2010 is very slow on a high speed Internet connection
2478286 (http://support.microsoft.com/kb/2478286)
Connection does not time out after inactivity time elapses in an OWA 2010 client connected to Exchange Server 2010 if published by using Forefront TMG 2010
2484988 (http://support.microsoft.com/kb/2484988)
A DNS server publishing rule stops working for a DNS server that is published by using Forefront TMG 2010
2478297 (http://support.microsoft.com/kb/2478297)
User Activity reports that are created by Forefront TMG 2010 show a wrong value in the reported data range
Notice that the first issue on this KB is the same that we were discussing on this TechNet thread here. So if you are facing such issue, make sure to install this update and run the script from KB2452980 (http://support.microsoft.com/kb/2452980). The other issue that we address on this rollup was raised from one of my customers as a problem, working with him I was able to reproduce the issue and after a long investigation we were able to find the root cause of the problem (in a great partnership with Exchange Team and TMG Developers), read KB2478286 (http://support.microsoft.com/kb/2478286) for more details. The third issue that we address on this TMG rollup is a DNS publication that stops to work, see KB2484988 (http://support.microsoft.com/kb/2484988) for more details. Last but not least a problem on the user activity report, simple stuff but that bothers for sure; see KB2478297 (http://support.microsoft.com/kb/2478297) for more details.
For ISA Server we are releasing the ISA Server 2006 hotfix package: December 2010, which includes the following updates:
KB article
2478307 (http://support.microsoft.com/kb/2478307)
MAPI client does not connect to an Exchange server on an internal network through an ISA Server 2006-based VPN connection on a computer that is running Windows 7
2481980 (http://support.microsoft.com/kb/2481980)
Unexpected authentication prompts while you use an OWA website that is published by using ISA Server 2006 SP1 if RSA authentication and FBA are used
Go get it and enjoy your holidays.
Merry XMas !!
As we just announced on Microsoft Press blog, three new Forefront eBooks are coming soon, check it out the full blog MS Press Blog. Here are they:
We hope you enjoy it.
Cheers !!
1. Introduction
This post is about a specific condition that can triggers the error 502 while browsing some web sites through TMG 2010 RC. The error message that the end users receives is similar to the one shown below:
The TMG Logging will not say much beyond as shown below:
This scenario was interesting because it was working sometimes but it didn’t work most of the time. Looking closely to the data I notice that when it works is because it hits one specific server in the destination web farm and when doesn’t work it hits another web server.
2. Understanding the Behavior
Using network monitor was possible to better understand why this happen:
1) The HTTP Header when it works:
- Http: Response, HTTP/1.1, Status Code = 200, URL: /
ProtocolVersion: HTTP/1.1
StatusCode: 200, Ok
Reason: OK
Date: Tue, 13 Oct 2009 15:57:06 GMT
Server: WEBSRV
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Cache-Control: private, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
P3P: CP="SRV"
Set-Cookie: reg_fb_gate=http%3A%2F%2Fwww.fabrikam.com%2F; path=/; domain=.fabrikam.com
Set-Cookie: reg_fb_ref=http%3A%2F%2Fwww.fabrikam.com%2F; path=/; domain=.fabrikam.com
Set-Cookie: test_cookie=1; path=/; domain=.fabrikam.com
Set-Cookie: lsd=zCI0G; path=/; domain=.fabrikam.com
Connection: close
TransferEncoding: chunked
+ ContentType: text/html; charset=utf-8
ContentEncoding: gzip
HeaderEnd: CRLF
+ chunkSize: 10
- ChunkPayload: HttpContentType = text/html; charset=utf-8
HtmlElement: ヒ
FooterEnd: CRLF
+ chunkSize: 8192
We can see that the HTTP response says that the following content will be chucked and after that response, destination server sends the other HTTP chucks:
WEBSRV TMG HTTP HTTP:HTTP Payload, URL: /
2) HTTP Header when doesn’t work
P3P: CP="WEBSRV2"
Set-Cookie: lsd=PQ6kd; path=/; domain=.fabrikam.com
X-Cnection: close
Date: Tue, 13 Oct 2009 15:38:13 GMT
ContentLength: 9970
+ payload: HttpContentType = text/html; charset=utf-8
Notice that the failing server don’t say that the content is chucked, however it is still sending more chucks after that:
WEBSRV2 TMG HTTP HTTP:HTTP Payload, URL: /
Since Chunked Transfer Encoding is a mechanism that allows HTTP messages to be split in several parts the first server is answering correctly, while the second server it is not. According to RFC if a server is using chunked encoding it must set the Transfer-Encoding header to "chunked". In order to compress the content we need to accumulate all the chucks and then compress. When it works, TMG knows that all that content is part of the same HTTP request since it says in the HTTP Response Header; therefore it waits for the entire content, compress and send it back to the client. On the failing server we receive the first answer that doesn’t say that the content is chucked and right after that we receive other chucks, since HTTP Compression is enabled it fails to reassemble all the content since it doesn’t know that they belong to the same content.
3. What can I do to fix it?
The best and most correct thing to do is to contact the administrator for the destination web server and report the problem; they should fix the issue since TMG is acting correctly. However if you want a workaround, that will be disabling the HTTP compression filter.
Hello Folks,
First I would like to thank you all for attending the DFW IT PRO Meeting yesterday (May 3rd) at Microsoft Las Colinas here in Irving, Texas. During our presentation we discussed the Concepts and Implementation of a Private Cloud Infrastructure using Windows Server 2012. As we said, the documentation to build your cloud infrastructure using Windows Server “8” Beta is already available at TechNet. You can read more about this documentation here. Many of you also said that never heard about the TechNet Wiki but do support the community based content. I would like you to read this post to better understand what the TechNet Wiki is all about.
Thanks again for your time yesterday and I hope to see you all again soon!
As announced last week at TechED North America, here it is the book cover for the upcoming book that me and Tom Shinder are writing for Syngress:
We are having an amazing time writing this book and we are looking forward to release it next year.
I had this article almost ready way back when I was on Forefront TMG Team but never had time to finish. This is about an issue where wspsrv.exe process was consuming high CPU in random moments of the day and the only workaround to make this process to use less CPU was to restart Firewall Service. Maybe the behavior sounds familiar, but the final resolution was never documented here in this blog.
2. Gathering Data
Using Process Monitor was possible to see that there were lots of ETW Trace threads running as shown below, which was kind of interesting to me:
To move forward in this investigation the usual perfmon and dump of the wsprv.exe process were collected while the issue was happening.
3. Analyzing the Data
Using the same approach that I documented in the Troubleshooting Forefront TMG 2010 Performance issues Cheat Sheet it was possible to notice a pattern in the threads that were stuck in Critical Section, all of them had a similar stack as shown below:
At that point it was clear to me that the component involved in such behavior was NIS, because is NIS that uses GAPA Engine (read NIS white paper for more information). As a test we disabled NIS and restarted Firewall Service and as a result of this action the issue stopped occurring.
4. Conclusion
Of course this was not the solution, as we don’t want to permanently disable this feature, but at least confirmed that NIS was the component causing the issue. We enabled NIS again and the issue came back. Another set of dumps and Process Monitor analysis lead the investigation to confirm that verbose tracing was enabled causing NIS to impact wspsrv.exe process by consuming more CPU. The traces are:
The value possible values are: 0, 1, 2, 3 and 4 corresponding to Error, Warning, Info, Function and Noise, respectively. In this case it was 4, which indeed caused a lot of noise. The resolution was to change back to zero and restart firewall service. It is important to clarify that is not always that this behavior will happen when the lower level trace is high, in order words, don’t think you can always repro this issue by just increasing this value. The issue was a combination of factors, in this particular scenario the server was very busy and by having the lower level trace so high the CPU utilization was increasing. The overall recommendation is to increase this value only for troubleshooting purpose and decrease after collecting data.
After Tom Shinder successfully implemented the contest quiz on his blog and give some prizes to the winner (Jason Jones) last month during the MVP Summit (I was there and saw how much Jason was happy ), I started thinking that I should follow my friend Tom on this cool initiative and do something similar. So here how it will work this contest:
Are you ready to play? Next Monday (March 28th) the first round of questions will come. Stay tune!!
The new URL Filtering option on Forefront TMG 2010 allows you to manually add web sites to a specific category; such feature is called URL Category Override. This can be a good approach when you want to force a specific site to be categorized in such way that it fits into a category that currently you have on your block rule. This post will describe a scenario where the TMG Administrator added a web site to the “dubious” category as shown below:
The goal was to block access to this web site due company security policy. To test if this configuration was working fine, TMG administrator used the Category Query feature and there it was possible to see that the new categorization was working fine as shown below:
When the client was trying to access this URL from his workstation he was getting the expected error message.
2. The Problem
The problem on this case is that users figured out a way to bypass this by typing https:// in front of the URL, in order words if they type https://www.facebook.com they were able to access the web site. You might be thinking: how is that possible? Well, that was my question when I first heard the TMG Administrator explaining his problem to me, but then after reviewing the environment and client configuration it was possible to understand why such behavior was happening.
The problem is that client workstation was configured as Secure NAT client, no web proxy configuration. You need to remember that URL Filtering doesn’t do HTTPS categorization for Secure NAT requests, therefore such behavior was expected. On the article that me, Jim and Mohit wrote for TechNet Magazine (March 2010 issue) we say:
“…the ability of URL filtering to evaluate the request is dependent on two criteria:
- Is the connection directed to the default HTTP port? If so, the Web proxy may be able to intercept this request and pass it to URL filtering for comparison. If not, the request will not be seen by URL filtering and thus cannot be compared to the database.
- If the connection is directed at the default HTTPS port, is HTTPS inspection enabled? If so, HTTPS inspection can bridge the connection, and URL filtering will have an opportunity to compare the request to the database.”
Based on that you can imagine how to fix this problem, correct? Let’s take a look on the options that we have here.
3. The Solution
In scenarios like this there are a couple of solutions:
- Enable HTTPS Inspection: with HTTPS Inspection enabled, it will be possible to enforce the URL Filtering for requests that use HTTPS and are coming from SecureNAT clients.- Use Web Proxy Client: by using web proxy client, URL Filtering will work regardless of the protocol.
For this particular scenario the administrator preferred to use Web Proxy Client and deploy a GPO to force all IE users to go out to the Internet using this particular TMG. For that the following AD policies were used:
Policy 1 – Used to force the proxy server setting:
Policy 2 – Used to disallow users to change their proxy settings
It is always important to analyze all the possible options and which one will best fit on your environment. Sometimes concentrate all the policy enforcement on the edge it is good, however there are many times on which you will need to make sure that your infrastructure as a whole is enforcing the company security policy. By leveraging Windows security capabilities to enforce policies you can facilitate the overall administration overhead and have multiple layers of policy enforcement in place.
Sometimes I receive questions like: I don’t want that user’s use the application XYZ to grab content on the Internet. How can TMG block this application on my Web Proxy Client? This is a classical question and it can be done on TMG if you have TMG Client installed, but if this is just a web proxy client, then the approach should be different. It comes back to the subject of enforcing company’s security policy end to end. Ask yourself the questions below and you will realize that there are much more to be concern about:
- Why this client is running a non approved application on company’s desktop in first place?- Why not use software restriction policy via GPO for the company workstations? - Even if you block on the edge, who guarantees that this non approved application is not trying to harm other internal clients?
As you can see there are many questions that need to be answered on this area before try to fix a particular non compliance concern by solely use a fix on the edge.
1. Another error 64?
After posting one of the reasons why ISA Server 2006 can come up with the generic error 64 in one of my posts, some readers asked me if this is the ultimate reason for this error. The answer is: it is not! Since the error 64 is generic it needs to be carefully interpreted, my previous post about this error mentions the “error 64” with the message: “host not available”.
This post will explain in more details why the error message below showed in the ISA Server 2006 Logging could occur while you are browsing Internet.
Figure 1 – Another error 64.
The error above was caught while the user was trying to browse www.fabrikam.com and download the Windows XP SP2 file. To simulate this problem I used the following lab:
Figure 2 – Lab used to simulate this problem.
2. Understanding the nature of this error
The 64: "The specified network name is no longer available" is a win32 error originally called ERROR_NETNAME_DELETED, this error is mapped in the winerror.h as:
//
// MessageId: ERROR_NETNAME_DELETED
// MessageText:
// The specified network name is no longer available.
In the network level, this problem could be cause by:
Network connectivity problems have various causes, but they typically occur because of incorrect network adapters, incorrect switch settings, faulty hardware, or driver issues. Some connectivity symptoms are intermittent and do not clearly point to any one of these causes.
Per KB325487.
Which means that is more under the TCP/IP level, which is controlled by the Windows OS rather than ISA Server itself.
3. Simulating the Problem
To simulate this problem I used a tool called Network Emulator for Windows and added high latency and random packet loss. Besides I also used the Web Application Stress Tool to add more load to my web server and really simulate a situation where server is busy. Now let’s take a look in the netmon trace got from the external interface of the ISA Server:
ISA Server sends the HTTP GET for the destination server:
12:39:13.355 192.168.1.113 192.168.1.95 HTTP HTTP:Request, GET /
- Http: Request, GET /
Command: GET
+ URI: /
Via: 1.1 ISACONTN1
If-None-Match: "304054985f13c91:4b2"
UserAgent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322)
Host: www.fabrikam.com
If-Modified-Since: Wed, 10 Sep 2008 16:09:25 GMT
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Accept-Language: en-us
UA-CPU: x86
Connection: Keep-Alive
Destination server sends the answer:
12:39:13.745 192.168.1.95 192.168.1.113 HTTP HTTP:Response, HTTP/1.1, Status Code = 200, URL: /
A HTTP GET is sent to get the XP SP2 file:
12:39:31.751 192.168.1.113 192.168.1.95 HTTP HTTP:Request, GET /XPSP2.zip
Destination server answers:
12:39:32.142 192.168.1.95 192.168.1.113 HTTP HTTP:Response, HTTP/1.1, Status Code = 200, URL: /XPSP2.zip
The file starts to be transferred:
12:39:32.242 192.168.1.95 192.168.1.113 TCP TCP:Flags=...A...., SrcPort=HTTP(80), DstPort=2050
12:39:32.242 192.168.1.113 192.168.1.95 TCP TCP:Flags=...A...., SrcPort=2050, DstPort=HTTP(80)
Suddenly the destination server resets the connection:
12:39:32.424 192.168.1.95 192.168.1.113 TCP TCP:Flags=.....R.., SrcPort=HTTP(80), DstPort=2050
12:39:32.584 192.168.1.95 192.168.1.113 TCP TCP:Flags=.....R.., SrcPort=HTTP(80), DstPort=2050
At this point the session was lost and the error showed in figure 1 appeared in the log.
What it is important for you after reading this post is to really understand that ISA Server for scenarios like this only externalize the problem. You need to focus on the real problem and start that by verifying:
· Which device is in between ISA and Internet?
o Don’t think that just because you have only a router in front of ISA Server that you will be “free of errors”. Routers do have updates and potential problems also.
· Can you sniffer the outside traffic to have the real picture of what comes into your network before hits the external interface of ISA Server?
o If you get the netmon trace only on the external interface of ISA and you have more devices in front of it you could be masquerading the real issue since you can’t see the clear traffic.
· If ISA is really the edge device, make sure that network interface card is update, the switch where ISA is connected is working properly, etc.
o Many administrators are only concern with updates on the OS level and forgot to address key updates do the drivers and active network devices.
Almost of the time the investigation of those errors occurs around ISA Server rather than in ISA Server itself. Keep your mind open to a broader set of possibilities instead of focus all our time and efforts in troubleshoot only ISA Server.
This week we launched ISA BPA V7 and this is a great opportunity to continue the explanation on how ISABPA can be useful for proactive and reactive work. Last session I explained how you can use ISA BPA for proactive work with ISA Server. This session will explain the benefits of using ISABPA while troubleshooting an issue.
2. ISA Data Packager
Besides the ISABPA itself, when you install this tool a group of programs is created within Microsoft ISA Server group as you can see in Figure 1:
Figure 1 – Tools that are installed by ISA BPA.
ISA Data Packager is a data gathering tool that can assist you to collect a set of data in one single shot. Let’s use as an example a scenario where user can’t access certain web sites. You can launch the ISA Data Packager and the first screen will present you the templates that are available:
Figure 2 – ISA Templates.
The template that you will choose will depend on the scenario that you are dealing with; here are some examples of usability of the main templates:
Scenario
Template
Unable to access Internet
Web Proxy and Web Publishing
Unable to access some parts of the web site
Get prompt for authentication when accessing a web site
OWA Exchange Publishing rule not working
Sharepoint Publishing rule no working
Unable to establish a VPN Site to Site
VPN
Unable to connect from a client to ISA using PPTP or L2TP
When I open my ISA Console I receive a 0x800 Error and nothing shows up
ISA Administration
ISA Console is crashing
Firewall Policy doesn’t show the rules
Monitoring / Configuration shows one node is not out of Sync
Configuration Storage Server
When accessing Internet Firewall Client turns red.
Firewall Client (NEW in ISABPA7)
Collect data from ISA Server to review the configuration later (no issue to reproduce)
Basic Repro and Static Configuration
Next step is to choose the template according to the scenario, for this example I’m going to chose Web Proxy and Web Publishing. After select and click Next you will see the following screen:
Figure 3 – Summary of the default selections
A set of options are selected by default when you choose the template, those options will vary according to the template that was previously selected. Notice that ISAInfo Report is not selected, which is something that is very useful since with this information you will be able to review all the details from this particular ISA box. In case you want to add that in your data collection you just need to click Modify Options and the following screen will appear:
Figure 4 – Changing default Options.
Here are some other guidelines about this screen:
· If you are having issues such as prompt for authentication when browsing internet, or ISA Server losing the secure channel with the DC, make sure to enable the option Netlogon Logging.
· If you are not dealing with Performance issue, disable the option Performance Monitor Snapshot.
· If you are using MSDE Database for logging and you want to collect data from it, select MSDE Error Logs.
· Change Tracking is NEW in ISABPA7.
After making the selection, click Start Data Collection and wait until the option press space bar to continue appears as shown in Figure 5:
Figure 5 – Starting capture.
At this point you should go to the workstation that is facing the problem and reproduce the issue that you are having. After reproduce the issue press space bar again in the collecting data window and wait until the CAB is generated.
3. Now What?
ISA Data Packager Creates a file called ISAPackage.CAB, by default located in the desktop. This file contains the following folder / files:
Folder
File(s)
Description
BpaDataPackagerLogFiles
BpaDataPackagerLogFile.txt
This is the ISA Data Packager Log that has information about the moment of the data collection. You will use this file to troubleshoot issues where the Data Packager failed to run for example.
IDP.2009-5-7.9-8-29.trace.log
Verbose logging for ISA Data Packager, also used to troubleshoot ISA Data Packager itself.
BpaReportFiles
BPAReport_ISACONTN1_200905070911.xml
This is the ISA BPA Health Check report that you can load using the ISA BPA Tool.
BPAReport_ISACONTN1_200905070911.xml.log
Log for data collection of the ISA BPA, used to troubleshoot ISA Health Check itself.
IsaConfigExport.200905070911.xml
This is the export configuration of the ISA Server. Consider this your backup if you never made one, here it goes.
EventViewerEvents
Application.evt
NEW on BPA7 – this is the export of the Application log in EVT format.
EventViewer_ErrorEvents6.csv
Only errors events (windows event viewer events) in CSV format.
EventViewer_IsaEvents6.csv
Only ISA errors events logged in windows event viewer.
System.evt
NEW on BPA7 – this is the export of the System log in EVT format.
IsaInfoFiles
ISAInfo_isacontn1.log
Log for data collection of the ISA Info, used to troubleshoot ISA Info itself.
ISAInfo_isacontn1.xml
ISAInfo file that you can open using ISA Info tool from ISA Tools .org.
ISALogs
IsaLogs_Firewall_TextEXT_200905070911.csv
Firewall Logging in CSV format.
IsaLogs_WebProxy_TextEXT_200905070911.csv
Web Proxy Logging in CSV format.
IsaTraces
isalog.bin
Files used by Microsoft CSS Engineers only since they require internal symbols to parse it.
manifest.txt
NetworkCaptures
External_20090507090839.cap
Network Capture from the external interface. Correct, you don’t need to start netmon separately when using ISA Data Packager.
LocalCorp_20090507090839.cap
Network Capture from the Internal interface.
Note1: amount of files and folders will vary according to the template that you choose.
Note 2: file name will vary according to the date of the day.
With this set of data you have enough data to start troubleshooting the issue that you are facing with ISA Server. You have logs, network captures and capability to read ISA Server configuration. My recommendation is that you install this tool in your lab, and start to test simple scenarios so you get used to read those logs. Try to simulate simple issues in your lab and look the logs to see what you can do to fix the issue.
One of the most challenges for the ISA Admin is to determine the culprit for an intermittent issue. This gets worse when the issue is related with performance. While there are many elements that can impact ISA Server’s performance, this post will describe an interesting case where the client was having problems to browse Internet through ISA Server. The web sites were coming up really slow and regardless of the browser (IE6 or IE7) the issue was happening.
2. Start from the Basics
Do not over estimate the basics, missing a basic check can cost you hours of deep troubleshooting while the monkey is right there looking for you. Here it is a checklist of things that should be reviewed:
· DNS Configuration: This can dramatically impact the performance if it is not correctly configured. Tom Shinder wrote a good post about an issue that he got because of the DNS that was not answering in time manner. Besides that, remember to review Microsoft recommendation on DNS configuration for ISA Server.
· RSS, Chimney and TCPCA: if you have Windows Server 2003 SP2 installed make sure to use KB936594 to address the issues that those keys can cause.
· Network Configuration: on ISA Server, make sure that the Network range is correctly defined. Review the article Troubleshooting Network Configuration in ISA Server.
· NIC Drivers: are the NIC drivers updated for the latest version?
· Speed and Duplex: autosense configuration between the Server and the Switch can also cause network intermittence behavior. Better force the speed (100 Mbps or 1Gbps) and duplex (full) in the switch port and in the NIC driver.
These are things that you can start looking in first hand. If all those elements look good than is time to move on and get more data.
3. The Output
After reviewing the netmon trace that was get it while the issue was happening the result was clear: did not have delay in the name resolution as well as the initial TCP handshake. The network communication was clear and there were not big gaps in between frames. The only thing that we notice was that it was taking too much time to transfer data to build the page. Something in the application level was not really going as good as it should.
The output from perfmon (using the counters from the article Monitoring and Troubleshooting Performance also did not show any suspicious activity. There was no leak in the server itself; processor utilization was good as well as memory and disk.
3. User Mode Dump
The next troubleshooting level was really to get a dump from wspsrv.exe and understand what that guy was doing during the page request. To do that we used the command below while repro the issue:
Cscript adplus.vbs -quiet -hang -pn wspsrv.exe
Note: for more information on how to use ADPLUS use the KB286350.
General Considerations:
· Install WinDbg that is part of the debugging tools.
· Configure the Symbol path using KB311503.
After load the dump in WinDBG we ran the command runaway to show the time that each thread is consuming:
0:000> !runaway
User Mode Time
Thread Time
21:f80 0 days 0:00:20.437
8:eb4 0 days 0:00:11.375
19:f78 0 days 0:00:02.953
25:fb8 0 days 0:00:02.656
42:10e8 0 days 0:00:01.859
45:152c 0 days 0:00:01.796
43:1318 0 days 0:00:01.718
35:1664 0 days 0:00:01.703
44:1534 0 days 0:00:01.562
…
The thread 21 is the one that seems to be using more resources. To see what this thread is doing run ~21kb, for this case the result was:
0:000> ~21kv
ChildEBP RetAddr Args to Child
0220f3a8 7c827d0b 7c83d236 000079e8 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
0220f3ac 7c83d236 000079e8 00000000 00000000 ntdll!NtWaitForSingleObject+0xc (FPO: [3,0,0])
0220f3e8 7c83d281 000079e8 00000004 646f30a0 ntdll!RtlpWaitOnCriticalSection+0x1a3 (FPO: [Non-Fpo])
*** ERROR: Symbol file could not be found. Defaulted to export symbols for W3Filter.dll -
0220f408 64709542 1adf1eec 1adf1eec 6470bead ntdll!RtlEnterCriticalSection+0xa8 (FPO: [Non-Fpo])
*** ERROR: Symbol file could not be found. Defaulted to export symbols for msfpc.dll -
WARNING: Stack unwind information not available. Following frames may be wrong.
0220f44c 615e7407 00000000 00000000 0220f470 W3Filter!CNetAddr::operator=+0x15e
0220f47c 615eb41d 202f3c28 0020f4a8 646f3080 msfpc!HmacMD5CreateKey+0x3944
0220f4ac 6470a167 1ad91218 202f3c28 0220f4d0 msfpc!TsLogInformationA+0x1b7
0220f4d4 6471bd0b 202f3c28 202f3558 202f3508 W3Filter!CNetAddr::operator=+0xd83
0220f508 647670c4 04e74008 00000000 00002faa W3Filter!DllUnregisterServer+0x1b4e
0220f520 6472676c 00002faa 64702748 00000002 W3Filter!DllUnregisterServer+0x4cf07
0220fdb0 647773ad 00000000 00000000 0000000a W3Filter!DllUnregisterServer+0xc5af
*** ERROR: Symbol file could not be found. Defaulted to export symbols for wspsrv.exe
0220fdf4 004c37d4 18b21558 00000001 00000000 W3Filter!DllUnregisterServer+0x5d1f0
0220fe3c 0046982d 00000001 00000001 00000000 wspsrv!IsNameInRwsConfigsLdt+0xb2af
0220fe7c 0046a570 00000001 00000000 20320c78 wspsrv+0x6982d
0220fee0 0046a8e0 00000109 00000000 00000001 wspsrv+0x6a570
0220ff04 0046a9e7 1d6ac2a8 0046a856 0220ff30 wspsrv+0x6a8e0
0220ff14 0046ab8b 00000109 00000000 00000001 wspsrv+0x6a9e7
0220ff30 00469939 00000109 00000000 00000001 wspsrv+0x6ab8b
0220ff50 00452a27 1d6ac34c 00000109 00000000 wspsrv+0x69939
0220ff7c 004536a4 1d6ac34c 00000109 00000000 wspsrv+0x52a27
Note: I’m using the public symbol server (SRV*c:\symbols*http://msdl.microsoft.com/download/symbols) which doesn’t have too much info about ISA Server symbols. For this reason I’m receiving the error showed above.
Let’s look at the critical section information:
0:000> !cs -s -l -o
-----------------------------------------
DebugInfo = 0x000cc228
Critical section = 0x000d0e74 (+0xD0E74)
LOCKED
LockCount = 0x6
WaiterWoken = No
OwningThread = 0x00000f74
RecursionCount = 0x1
LockSemaphore = 0x7398
SpinCount = 0x00000000
OwningThread DbgId = ~18s
OwningThread Stack =
0214fd94 7c827d0b 77e61d1e 00000d38 00000000 ntdll!KiFastSystemCallRet
0214fd98 77e61d1e 00000d38 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
0214fe08 77e61c8d 00000d38 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
0214fe1c 0043b9fd 00000d38 ffffffff 000d0e08 kernel32!WaitForSingleObject+0x12
0214fe6c 615e7407 00000000 00000000 0214fe90 wspsrv+0x3b9fd
0214fe9c 615eb41d 1dbea130 0014fec8 0041e4f8 msfpc!HmacMD5CreateKey+0x3944
0214fecc 004d3686 000abb98 1dbea130 0214fef4 msfpc!TsLogInformationA+0x1b7
0214ff04 0042beec 1dbea130 1dbea130 1dbea108 wspsrv!GetNetworkVIPAddress+0x66e5
0214ff20 00435549 00000000 00000001 1dbeaa94 wspsrv+0x2beec
0214ff50 0042b850 1dbeab3c 0214ff74 00430f1c wspsrv+0x35549
0214ff5c 00430f1c 00000000 6370a830 1dbeaa94 wspsrv+0x2b850
0214ff74 63705056 00000000 00000000 00409960 wspsrv+0x30f1c
0214ff94 00453616 00000001 00000000 00000000 ratlib!RatPollTimer+0x1a9
0214ffb8 77e64829 00000001 00000000 00000000 wspsrv+0x53616
0214ffec 00000000 00453576 00000001 00000000 kernel32!BaseThreadStart+0x34
DebugInfo = 0x000a8890
Critical section = 0x000d0e38 (+0xD0E38)
LockCount = 0x0
OwningThread = 0x000006dc
LockSemaphore = 0x0
OwningThread DbgId = ~37s
2294f96c 7c827d0b 71b21af5 0000890c 00000001 ntdll!KiFastSystemCallRet
2294f970 71b21af5 0000890c 00000001 2294f998 ntdll!NtWaitForSingleObject+0xc
2294f9ac 71b2c517 0000890c 00008fc4 00000000 mswsock!SockWaitForSingleObject+0x19d
2294fa24 71c094e5 00008fc4 2294fa84 00000001 mswsock!WSPRecv+0x203 (FPO: [Non-Fpo])
2294fa60 71bb1151 00008fc4 2294fa84 00000001 ws2_32!WSARecv+0x77 (FPO: [Non-Fpo])
2294fa8c 6d561686 00008fc4 00000000 00001000 wsock32!recv+0x31 (FPO: [Non-Fpo])
2294fadc 4e2597ce 00007530 012bdd40 012bdd40 dbnetlib!ConnectionRead+0x3b6 (FPO: [Non-Fpo])
2294fb10 4e25982d 012b0450 012bdd40 00000009 sqloledb!CDataSource::ConnectionRead+0x35
2294fb5c 4e252358 01710cc6 00000001 00000000 sqloledb!CDBConnection::GetBytes+0x269
2294fba8 4e2555c4 01158560 00000088 0000001e sqloledb!CDBConnection::ProcessTDSStream+0x157
2294fc64 4e255691 01155e78 00000049 01157688 sqloledb!CStmt::ExecDirect+0x786
2294fc7c 4e254d32 01155e78 00000049 00000000 sqloledb!CStmt::SQLExecDirect+0x28
2294fcac 4e25517d 00000000 4e25321c 00000049 sqloledb!CCommand::ExecuteHelper+0x157
2294fd30 4e254c4b 01157688 00000000 615d30b0 sqloledb!CCommand::Execute+0x76b
2294fd68 6160de22 0114dd78 00000000 615d30b0 sqloledb!CImpICommandText::Execute+0xdd
2294fd98 6160e2a1 00000000 00000000 2294fdfc msfpc!CFastSession::Insert+0xe2
2294fdb4 6160e776 201f8bf0 2294fe14 2294fdfc msfpc!SessionInfo::~SessionInfo+0x4f
2294fee0 6160e9f5 201f8bf0 0115a278 615d1a14 msfpc!OpenFastLoadRowset+0x4cb
2294ff08 6160eaa4 00000000 000d0010 6160ecd1 msfpc!CFastSession::Commit+0x29
2294ff20 0050a069 00000001 004d78e6 000d0010 msfpc!CFastSession::~CFastSession+0x18
The red line in the second stack shows that the machine is submitting a SQL statement using the SQLExecDirect function. Now let’s see what SQL command is being executed:
0:035> du 01155e78
01155e78 "EXEC sp_batch_insert [##Firewall"
01155eb8 "Log000000391SDCTSI00FW02], [Fir"
01155ef8 "ewallLog]"
Ok, now things start to make sense. The problem was happening during that time and the ISA Server Firewall Service was writing to the log file (Firewall Log) which was located in a SQL Server database.
4. Logging was the Problem
The browsing performance issue for this case was caused by a performance problem in the SQL Server. The SQL Server that ISA Server was using for logging had disk I/O problems and therefore it was very slow to answer to network requests. The workaround for that while customer fixed his SQL issue was to change the logging type for TXT in the local machine.
The browsing experience improved tremendous and this is actually expected. You can review the ISA Server Logging Best Practices and you will see that TXT log format has the best performance in comparison with the other log types.
5. Additional Reading - Learning More about Debugging
Here are good references for learning more about debugging:
Books
· Advanced Windows Debugging by Hewardt and Pravat
o This book has two great authors from Microsoft and it also foreword by Mark Russinovich, so you know that it is really worth to have it.
· Memory Dump Analysis Anthology, Volume 1 by Dmitry Vostokov
o This book is very good to startup on debugging. It really starts from the basics and it keeps going more in depth. Easy to read and very didactic.
· Windows Internals by Mark Russinovich (Author), David A. Solomon (Author)
o This should be a book that every IT professional should have on his desk if he really wants to know how Windows Architecture works. Just amazing.
Online Resources
· One of the best sites that I found about dump analyses: http://www.dumpanalysis.org/blog/ maintained by Dmitry Vostokov
· There are two really good blogs from two personal friends from Microsoft:
o http://blogs.msdn.com/debuggingtoolbox/ from Roberto Farah (Sharepoint PFE).
o http://blogs.technet.com/marcelofartura/ from Marcelo Fartura (IIS PFE).
· Very good in depth online resource: http://blogs.msdn.com/tess
· Microsoft Advanced Windows Debugging and Troubleshooting: http://blogs.msdn.com/ntdebugging
Once again, ISA Server was only a victim of an environmental issue J !!
Although the VPN template screen (see figure below) doesn’t seems to have any news on this area, the new TMG Data Packager introduces new logs that can assist you when troubleshooting VPN site to site issues.
The Oakley log file that TMG Data Packager creates contains the IKEEXT.ETL (IKE Tracing) and the WFP.TMF (file that will be used to parse the ETL file) files. In order to parse this file you will need to download the tools TRACEFMT.exe and TRACEPRT.dll from the Windows XP SP2 Support Tools. After installing those tools you can extract the content of the TMG CAB file to a folder and run the command below to parse it:
C:\Program Files\Support Tools>tracefmt.exe Y:\temp\VPN\TmgPackage\IkeExt\ikeext.etl -tmf Y:\temp\VPN\TmgPackage\IkeExt\wfp.tmf -o Y:\temp\IKEOutput.txt
Setting log file to: Y:\temp\VPN\TmgPackage\IkeExt\ikeext.etl
Getting guids from Y:\temp\VPN\TmgPackage\IkeExt\wfp.tmf
Event traces dumped to Y:\temp\VPN\TmgPackage\IkeExt\IKEOutput.txt
Event Summary dumped to Y:\temp\VPN\TmgPackage\IkeExt\IKEOutput.txt.sum
Exit Status: 38
After converting it you can read the IKEOutput.txt file, there you will find the log in the following format:
Package is received and processed according to IPSec Parameters that should match between both endpoints:
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 0|192.168.0.10|Received packet
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 0|192.168.0.10|Local Address: 192.168.0.7.500 Protocol 0
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 0|192.168.0.10|Peer Address: 192.168.0.10.500 Protocol 0
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|iCookie 98b22fe79d9d675f rCookie 1610c0b30c6bbe60
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|Exchange type: IKE Quick Mode Length 300 NextPayload HASH Flags 1 Messid 0x3d6edc77
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|mmSa: 0x000000000265B9F0
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|Create QMSA: qmSA 000000000265ED60 messId 3d6edc77
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|Processing QM. MM 000000000265B9F0 QM 000000000265ED60
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|Process Payload HASH, SA 000000000265B9F0 QM 000000000265ED60
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|Process Payload ID, SA 000000000265B9F0 QM 000000000265ED60
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|Process Payload SA, SA 000000000265B9F0 QM 000000000265ED60
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|QM propNum 1, transformNum 0, peerSpi 3151228040
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|QM transNum 1
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|PROTO: ESP Algo 3
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|IPSEC_LIFE_TYPE: 1
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|IPSEC_LIFE_DUR: 3600
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|IPSEC_ENCAPSULATION_MODE: 1
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|IPSEC_HMAC_ALG: 2
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|IPSEC_GROUP_DESC: 2
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|IsRecvPolicyTunnelPolicy: TRUE
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|Looking up QM policy for IKE
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|QM localAddr : 10.10.10.0.0 Mask 255.255.255.0 Protocol 0
[0]00F8.0B50::01/01/1601-05:01:53.375 [ikeext] 1|192.168.0.10|QM peerAddr : 10.40.40.0.0 Mask 255.255.255.0 Protocol 0
Policy identification and verification
[0]00F8.0B50::01/01/1601-05:01:53.387 [ikeext] 1|192.168.0.10|Policy
GUID: {a167bf6c-78ff-4b3d-b619-1ea03d29664a}
LUID: 0x8000000000000003
Name: ISA VPN S2S tunnel to network STSTMG
Description: (null)
Flags: 0x00000000
Provider: <unspecified>
Provider data:
Verification of the Quick Mode parameters
Type: IKE Quick Mode Tunnel
Proposals: 1
-- 0 --
Lifetime:
Seconds: 3600
Kilobytes: 100000
Packets: 2147483647
PFS group: 2
SA transforms: 1
Type: ESP-Auth & Cipher
Auth transform:
Type: SHA1
Config: HMAC-SHA1-96
Crypto module: <unspecified>
Cipher transform:
Type: 3DES
Config: CBC-3DES
Flags: 0x00000080
Dont negotiate 'byte' lifetime
Local tunnelEndpoint: 192.168.0.7
Remote tunnelEndpoint: 192.168.0.10
Normal idle timeout (seconds): 300
Idle timeout in case of failover (seconds): 60
.
. log continues..
The log can be pretty extensive and it is very important to know what you are looking for (which error are you chasing), mainly when the scenario is related to TMG site to site VPN with third party vendors. Sometimes the IPSec parameters doesn’t match and this is the most common cause for failures during the IPSec negotiation. This logging can be pretty handy in those scenarios since it gives verbose information about what it is happening behind the scene.
Microsoft released the Security Compliance Manager version 1.0 beta, this tool provides centralized security baseline management features, baseline portfolio, customization capabilities, and the ability to automate the creation of GPOs and System Center Configuration Manager DCM packs. Here are some features and benefits of this tool:
To participate on the Beta for the application register at https://connect.microsoft.com/site715
Download is available at https://connect.microsoft.com/site715/Downloads/DownloadDetails.aspx?DownloadID=20423
More info about others security guidance go to http://technet.microsoft.com/en-us/library/cc184906.aspx
The Update 1 Rollup 3 for Microsoft TMG 2010 is now available for you. This rollup address the following issues:
2501646
FIX: "A security package specific error occurred" error when you run a recurring report on a Forefront TMG 2010 server that is managed by an EMS and that is in a workgroup
2502685
FIX: "0xc0360007 (STATUS_IPSEC_CLEAR_TEXT_DROP)" error when you try to access the internal IP address of a Forefront TMG 2010 server through an IPsec site-to-site network
2472894
"HTTP/1.1 502 - Error 11 Bad format" error when you access SSL websites that use SAN certificates in Forefront TMG Server 2010 if a non-English version of a Windows operating system is installed
2501650
FIX: "Page Cannot Be Displayed" error when you try to access a website that requires a client certificate authentication on a Forefront TMG client in Forefront TMG 2010 if HTTPS Inspection is enabled
2501776
FIX: "502 Proxy Error. An attempt was made to load a program with an incorrect format. (11)" error when you try to use a HTTPS URL through Forefront TMG 2010 if HTTPS inspection is enabled
2498831
How to configure the "HTTPS inspection caching in a forward proxy scenario" and "HTTPS inspection inclusion list" features in Forefront TMG 2010
2498837
An enterprise node is incorrectly added in Forefront TMG MMC after you install Forefront TMG 2010 SP1 Update 1
2445386
"Sign in as a Different User" does not work on a SharePoint website that is published by Forefront TMG 2010
2498835
PPTP or L2TP/IPsec connection is not reestablished between Forefront TMG 2010 servers
2501777
FIX: "502 Proxy Error. An unknown error occurred while processing the certificate. (-2146893017)" error when you try to access a website over HTTPS in Forefront TMG 2010 if HTTPS inspection is enabled
2497959
Forefront TMG Firewall service may stop when users run desktop sharing software over HTTPS that is proxied by Forefront TMG 2010
2500737
"0xc0040446" or "0xc004041d" error if the primary IP address or DNS address uses 128.0.0.0/16, 191.255.0.0/16, or 223.255.255.0/24 in Forefront TMG 2010
2497858
SCOM logs many "Forefront TMG Server - Cache: Current Cache Fetches Average Ms Per Request error" error alerts from TMG Management Pack through Forefront TMG 2010
2501755
Mspadmin.exe may crash if you do not use SQL Server Express to log traffic in Forefront TMG 2010
2502686
Forefront TMG Firewall service might crash when WP_TRAFFIC tracing is enabled in Forefront TMG 2010
2501782
"0xc004039E" error when you use the "Allow user override" setting for a HTTP deny rule in an enterprise policy in Forefront TMG 2010
2501780
FIX: Forefront TMG Job Scheduler service (Isasched) stops responding on an array member server that is not a report server in Forefront TMG 2010
As you can see there are a lot of fixes in this rollup, I particularly worked in many issues involving 2501650 and 2502686 while the hotfixes were not even ready. Due the nature of those issues I strong recommend you to download this update and plan the installation on your Forefront TMG. To install this update, you must have TMG 2010 SP1 and Update 1 already installed.
Got get it at http://support.microsoft.com/kb/2498770.
E-Mail Protection feature in TMG was a feature that I used to work a lot when I was in CSS. This year I delivered a session internally about Troubleshooting E-Mail Protection in TMG and today I’m sharing the slide deck (the public version) with you. Feel free to download it:
Note: most of the troubleshooting tips that I added in this slide deck were also included in the Forefront Threat Management Gateway (TMG) 2010 Troubleshooting Survival Guide, E-mail Protection Troubleshooting section.
Enjoy it
Introduction
Recently I was engaged on this TechNet Forum thread and I thought that it was an interesting question and concern. Let me explain the scenario on which I was able to repro the behavior on my own lab:
Contoso has a security policy where only users that belong to the Internet Users group can access Internet. To enforce that they created a rule on ISA Server on which only members of the Internet Group can access Internet. Contoso uses Internet Explorer 6 on all workstations of their network. The process today is that once the user receives authorization to access Internet, the AD Admin adds the user to Internet Users group and the user just need to open a new browser session and he will have access. Recently Contoso upgraded 50% of their workstations to use Internet Explorer 8, they notice the following behavior since this change: if an user that doesn’t belong to Internet Users group is already logged on the workstation and the administrator add this user to the Internet User’s group, the user receives an error when try to browse Internet, the error says that ISA denied the request. If the user logs off from the windows and logon again it works fine.
Why this is happening?
What it is happening on this case is simply the way that Internet Explorer 8 works for the authentication part of the conversation. Since the version 7, Internet Explorer was capable to use Kerberos for proxy authentication (while IE6 and lower only use NTLM and Basic). ISA Server will negotiate the authentication with the browser and will authenticate according to the method that is supported on both sides. The difference in this case is the following:
With IE7 (and higher)
1. Client sends the GET request to www.microsoft.com (for example). This request goes as anonymous.
2. ISA will send the 407 asking for authentication.
3. If client has already a Kerberos cached ticket it will not go to the DC to get a new one, if there is no ticket then it will go to the DC to get an updated ticket to send to ISA.
4. Client will send another GET Request now with the credentials and the ticket.
5. ISA will verify the request and allow (or deny according to the rule).
Note: in this case since ISA doesn't go to the DC, it relies on user's token to access the resource. Since the current ticket doesn’t include the update on user’s group membership, ISA will verify that the user doesn't below to the group and will deny the request.
With IE6
3. Client sends another GET request with the credentials (NTLM).
4. ISA goes to the DC to authenticate the user.
Note: since ISA goes to the DC it will get an updated version of the user's group membership and will verify that the user now belongs to this group.
5. ISA will allow the user to pass through.
Under the Hood
Prior to get to this conclusion I decided to do some tests to validate the theory and the tests worked pretty fine. Here it is the traffic from the client workstation with the user “Yuri” logged in, right after the inclusion of the user to Internet User’s group:
1. Client sends the GET Request
10.20.20.201 10.20.20.1 HTTP HTTP:Request, GET http://www.microsoft.com/
2. ISA asks for authentication:
10.20.20.1 10.20.20.201 HTTP HTTP:Response, HTTP/1.1, Status Code = 407, URL: http://www.microsoft.com/
Connection: Authentication
StatusCode: 407, Proxy authentication required
Reason: Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy filter is denied. )
+ ProxyAuthenticate: Negotiate
Proxy-Authenticate:
+ ProxyAuthenticate: Kerberos
+ ProxyAuthenticate: NTLM
Connection:
ProxyConnection: Keep-Alive
Cache-Control: no-cache
+ ContentType: text/html
ContentLength: 4113
+ payload: HttpContentType = text/html
Note: Notice that ISA is negotiating the authentication method.
3. Client sends another GET request now with the credential.
10.20.20.201 10.20.20.1 HTTP HTTP:Request, GET http://www.microsoft.com/ , Using SPNEGO Authorization
+ URI: http://www.microsoft.com/
Accept: */*
UserAgent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322)
Accept-Encoding: gzip, deflate
Cookie: MUID=42623805587D4F6EB894604864D99E47; WT_FPC=id=173.74.31.197-401582496.30090674:lv=1279361835221:ss=1279361835221; MC1=GUID=ba325a1581e1aa4dbcd0988320053feb&HASH=155a&LV=20107&V=3; A=I&I=AxUFAAAAAABDBgAAgb+LoGxG8UwjOO2L0QId7Q!!
- ProxyAuthorization: Negotiate
- Authorization: Negotiate YIIE5QYGKwYBBQUCoIIE2TCCBNWgJDAiBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKwYBBAGCNwICCqKCBKsEggSnYIIEowYJKoZIhvcSAQICAQBuggSSMIIEjqADAgEFoQMCAQ6iBwMFACAAAACjggO7YYIDtzCCA7OgAwIBBaEOGwxDT05UT1NPLk1TRlSiKTAnoAMCAQKhIDAeGwRIVFRQGxZpc2Fjb250bjEuY
WhiteSpace:
- NegotiateAuthorization:
Scheme: Negotiate
- GssapiKrb5: 0x1
Kerberos:
Note: Notice that client is using Kerberos.
4. This ticket doesn’t contain an updated information about user’s group membership, hence ISA Server sends a deny.
10.20.20.1 10.20.20.201 HTTP HTTP:Response, HTTP/1.1, Status Code = 502, URL: http://www.microsoft.com/
StatusCode: 502, Bad gateway
Reason: Proxy Error ( The ISA Server denied the specified Uniform Resource Locator (URL). )
ProxyConnection: close
ContentLength: 4059
5. At this point, I ran the command klist on the workstation and here are the tickets that I had at that point:
C:\Program Files\Windows Resource Kits\Tools>klist tickets
Cached Tickets: (3)
Server: krbtgt/CONTOSO.MSFT@CONTOSO.MSFT
KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)
End Time: 7/17/2010 18:34:46
Renew Time: 7/24/2010 8:34:46
Server: HTTP/isacontn1.contoso.msft@CONTOSO.MSFT
Server: host/client1.contoso.msft@CONTOSO.MSFT
6. I thought to myself: well, if I wipe all the tickets I will force the user to go to the DC and get an updated ticket, this way if I try again to access Internet, I will use the updated ticket. To do that I ran the command: klist purge. I deleted all tickets and the end result was:
C:\Program Files\Windows Resource Kits\Tools>klist purge
Cached Tickets: (0)
Now is time for the truth, tried to access Internet again and then it WORKED. Here it is the netmon trace for this second attempt with my Kerberos ticket cache clean:
3. Client goes to the DC to get ticket for the user “Yuri”:
10.20.20.201 10.20.20.20 KerberosV5 KerberosV5:AS Request Cname: yuri Realm: CONTOSO.MSFT Sname: krbtgt/CONTOSO.MSFT
- Kerberos: AS Request Cname: yuri Realm: CONTOSO.MSFT Sname: krbtgt/CONTOSO.MSFT
- AsReq: Kerberos AS Request
+ ApplicationTag:
+ KdcReq: KRB_AS_REQ (10)
4. The DC responds:
10.20.20.20 10.20.20.201 KerberosV5 KerberosV5:AS Response Ticket[Realm: CONTOSO.MSFT, Sname: krbtgt/CONTOSO.MSFT]
- Kerberos: AS Response Ticket[Realm: CONTOSO.MSFT, Sname: krbtgt/CONTOSO.MSFT]
- AsRep: Kerberos AS Response
+ KdcRep: KRB_AS_REP (11)
5. Client goes again to the DC to get a ticket to the HTTP Service (ISA SPN):
10.20.20.201 10.20.20.20 KerberosV5 KerberosV5:TGS Request Realm: CONTOSO.MSFT Sname: HTTP/isacontn1.contoso.msft
- Kerberos: TGS Request Realm: CONTOSO.MSFT Sname: HTTP/isacontn1.contoso.msft
- TgsReq: Kerberos TGS Request
+ KdcReq: KRB_TGS_REQ (12)
6. The DC responds:
10.20.20.20 10.20.20.201 KerberosV5 KerberosV5:TGS Response Cname: Yuri
- Kerberos: TGS Response Cname: Yuri
- TgsRep: Kerberos TGS Response
- KdcRep: KRB_TGS_REP (13)
+ SequenceHeader:
+ Tag0:
+ PvNo: 5
+ Tag1:
+ MsgType: KRB_TGS_REP (13)
+ Tag3:
+ Crealm: CONTOSO.MSFT
+ Tag4:
+ Cname: Yuri
+ Tag5:
+ Ticket: Realm: CONTOSO.MSFT, Sname: HTTP/isacontn1.contoso.msft
+ Tag6:
+ EncPart:
7. Client sends the GET request to ISA now with the new ticket:
Host: www.microsoft.com
- Authorization: Negotiate YIIFDQYGKwYBBQUCoIIFATCCBP2gJDAiBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKwYBBAGCNwICCqKCBNMEggTPYIIEywYJKoZIhvcSAQICAQBuggS6MIIEtqADAgEFoQMCAQ6iBwMFACAAAACjggPjYYID3zCCA9ugAwIBBaEOGwxDT05UT1NPLk1TRlSiKTAnoAMCAQKhIDAeGwRIVFRQGxZpc2Fjb250bjEuY
8. ISA allows the traffic:
10.20.20.1 10.20.20.201 HTTP HTTP:Response, HTTP/1.1, Status Code = 302, URL: http://www.microsoft.com/
Conclusion
As you could see on this post this is not really an issue, neither on ISA nor on Internet Explorer, this is actually a behavioral change that Internet Explorer 7 and higher had. This is a good improvement, because Kerberos is much lighter and doesn’t put too much pressure on the DC. One classic problem with NTLM is that the authentication request against the DC is huge, read the article Improving Web Proxy Client Authentication Performance on ISA Server 2006 that I wrote couple of years ago and you will see the advantage that IE7 has over IE6.
But, if it is a requirement for your business that changes to group membership immediately reflect on user’s Internet browsing experience than you can use ISA’s IP address on the IE setting rather than the name, this way you are forcing to use NTLM rather than Kerberos. Just keep in mind that when too much pressure is added on the DC you can face the infamous “random prompt for authentication” on ISA due the 5783 and 5719 events.
You might not have noticed but this month (last January 10th) ISA Server 2006 Mainstream Support ended as shown in the table below:
Source: http://support.microsoft.com/lifecycle/?p1=11928
The question that you might have is: what about this extended support that goes until January 2017? Extended support means the following:
The Extended Support phase follows Mainstream Support for Business and Developer products. At the supported service pack level, Extended Support includes:
Source: http://support.microsoft.com/lifecycle/
Better to start planning your migration as part of your new year’s resolution.
This post is about a problem where Outlook was working fine through TMG publishing rule, however when TMG Admin tried to access OAB and OOF through Outlook he got an error. To bypass Outlook he tried to access https://mail.contoso.com/ews/exchange.asmx and got 403. The 403 was coming from Exchange vdir /EWS/, here an example of the header:
10.20.20.11 10.20.20.1 HTTP HTTP:Response, HTTP/1.1, Status Code = 403, URL: /ews/ - Http: Response, HTTP/1.1, Status Code = 403, URL: /ews/ ProtocolVersion: HTTP/1.1 StatusCode: 403, Forbidden Reason: Forbidden Server: Microsoft-IIS/7.5 Set-Cookie: exchangecookie=599fc2a7540e4e66b1169d9d5c358aa5; expires=Sat, 17-Jul-2011 21:39:05 GMT; path=/; HttpOnly XPoweredBy: ASP.NET Date: Fri, 29 Jan 2010 21:39:05 GMT ContentLength: 0 HeaderEnd: CRLF
Resolution: after some investigation we notice that the /EWS has anonymous on it (/EWS vdir on Exchange 2007 doesn't have anonymous by default), after disabling anonymous and leave only Basic (to match with the delegation) it worked.
Important points before adopting this resolution:
While working on this issue with the Exchange folks they warned me about this action (disabling anonymous for /EWS on Exchange 2010) and they told me that:
“There are some issues if you disable anonymous on /EWS/ vidr for Exchange 2010. Anonymous is enabled on the virtual directory because EWS uses ws-security for federating calendars and free/busy across organizations for the new calendar sharing feature. Federation occurs via the ws-security protocol, which authenticates via SOAP <wssecurity> header rather than an HTTP authentication header. IIS must let such requests go through, after which WCF (upon which EWS is built) will properly authenticate them - in other words the "anonymous" IIS setting does not allow anonymous requests to get through to EWS. Turning off anonymous has some side effects, namely that cross-organization (federated) calendar sharing breaks as does federated mailbox migration.”
Having those considerations in mind, what you can do in TMG to overcome that without disabling anonymous is:
This week Microsoft released a major update of Forefront TMG 2010 and many TMG Admins are very excited with the new features that were announced in the Forefront TMG team blog, such as the support for Kerberos authentication in an array scenario, the improved error pages and the new site activity report. These are already three reasons to apply SP2 on your TMG, but instead of only adding two other features I’m going to give you five more reasons to apply this update. Here are those:
1. Forefront TMG 2010 SP2 makes TMG startup operation ten times faster.
2. Do you remember KB2498831? No need to run that script anymore, with TMG 2010 SP2 a new option was added in the screen to allow you to do that as shown below:
3. Performance improved for cloud migration.
4. Improvement in the E-Mail Protection feature
5. Account lockout enhancements for FBA.
That’s it…go grab TMG 2010 SP2 and remember: in order to apply TMG 2010 SP2 you need TMG 2010 SP1 + Update 1.
If you are following my blog for a long time you probably read the post TMG E-Mail Protection Feature x Exchange 2010 SP1 (first published more than an year ago) when we were dealing with a major E-Mail protection issue on TMG. Due the nature of the integration between Forefront TMG and E-Mail Protection feature (Forefront for Exchange and Exchange Edge) I also wrote this presentation to assist you while troubleshooting this feature.
The good news is that Forefront TMG 2010 SP2 brings to you the following fixes that will alleviate lots of the issues that were present in the past with this integration:
Go get SP2 and enjoy it!!
One of the presentations that I delivered this year at TechED Brazil was about On-Premise Security while Migrating to the Cloud. There are many reasons to migrate to the cloud and during this presentation I emphasized the three core elements below:
New Economics
While those core elements sounds very good, we must also be alert for the new challenges that comes with this adoption, such as:
New Threat Landscape
The presentation was really focus on the second bullet (on-premise security). Some of the reasons why this is still an important point to address include:
The misconception that the migration to the cloud means offloading your security to the cloud provider is just plain wrong. You need to be diligent because at the end of the day it is your data that could get compromise if you relax the on-premise security. You should adopt a defense in depth approach. All the elements from the endpoint to the cloud must be secure, not only the hosts, but the path and the remote clients. Here is a typical example of how this will look like:
There are five key elements in this diagram
In summary the path to the cloud requires a lot of planning to make sure that your users can have a seamless experience while you keep your data secure.