One thing that most of the Firewall’s administrators struggle with is how to provide a secure outbound control without hurting the user’s experience. Users don’t want to receive an error saying “unable to access the page” or a straight forward “access denied”. Users want a better feedback about what is going on and why they can’t access the web site that they thought it was okay to access.
TMG enhances this experience by allowing you to customize the error message from one single place….yeah, that’s pretty cool. Now when you can create a Deny rule and instead of redirect to another URL (which you can still able to do) you can type in there which error messages you want that the user receives as shown in Figure 1:
Figure 1 – Simple, easy and effective way to give a feedback to the end user.
On the user’s side what he will receive is a window similar to Figure 2:
Figure 2 – User’s experience is improved with a friendly error message.
You haven’t play with TMG Beta 2 yet? Time to start playing since Beta 3 is on the way for this summer. Got get Beta 2 at:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=e05aecbc-d0eb-4e0f-a5db-8f236995bccd
Introduction
Sysinternals tools are just amazing to troubleshoot a huge amount type of issues: networking, AD, core OS, etc. But, one thing that many security administrators don’t realize it yet is that those tools also are great to troubleshoot ISA Server issues in different scenarios. Before move on to the tool that I want to talk about, here are some other articles that I wrote where Sysinternals tools were used:
Article: Firewall Client is Unable to Connect to ISA Server 2006
Tool: FileMon
Article: Unable to open a link for a MHT file using Internet Explorer 6 through ISA Server 2006
Article: Hardening ISA Server in a Supported Manner
Tool: Regmon
Article (Portuguese): O Internet Explorer executou uma operação ilegal e será fechado
Tool: Process Explorer
Now, let’s play with a cool tool called: ADInsight.
Using ADInsight to Track ADAM Calls
ADInsight is a tool that allows you to view LDAP calls on the fly from a nice GUI interface. For the purpose of this example I’m going to follow the steps below to generate some data:
1. Launch ADInsight
2. Launch ISA Server 2006 Management Console
3. Review the data created by this operation
As soon as we execute step two ADInsight starts to collect information and notice in the sample below that the process is still MMC.exe but it is already accessing the local ADAM on port 2171:
Figure 1 – Initial information when launch ISA Management Console.
After complete load the ISA Server Management console the process changes and if you click in one of the lines in the upper pane and the lower pane will show in more details the parameters that were used during that operation:
Figure 2 – LDAP information with more details in the lower pane.
Now you can dig in and see more what’s going on behind the scenes. Enjoy it !!
Note: If you want to learn more about Sysinternals tools, read Windows® Sysinternals Administrator's Reference (Inside Out) by Mark Russinovich.
Just this week I received around two cases where administrators want to access OWA through their mobile devices. Not Active Sync, not OMA….OWA. From the Exchange Team perspective this is not supported and the article below explains list the browser’s version and Windows’s version are supported by OWA:
Educating Information Workers About Outlook Web Access
http://technet.microsoft.com/en-us/library/aa998931.aspx
1. Introduction
Last post I was mentioning how important it is to have a good planning phase for a technology that you are going to implement it. Planning doesn’t mean only sizing the right hardware for your needs, but also how you will configure the Server to be capable to handle the traffic that your network needs. Sometimes changing the default configuration is necessary, but those changes also need to be planned.
2. Scenario
The example that inspired this post comes from a real scenario where the Firewall Administrator changed the default memory used for cache (which is 10%) to 40% (in a 1GB RAM Server scenario), as showed in Figure 1.
Figure 1 – Default memory used for cache.
As result what was initially done with good intentions (because they want to cache more) caused the Internet access downtime since the server started to run out of resources. The big argument was: the environment was running just fine for at least one year, why just now the issue is happening? There are many reasons for that, such as:
· Amount of users grow over the year.
· Amount of Internet access increased.
· Amount of applications that need to have Internet access increased.
If you don’t know the answer for those questions you will need at least a traffic profile plus a performance baseline of the environment when was working fine to them compare to a scenario that fails.
In this scenario, when the server was running out of resource we got the ISA Data Packager in repro mode plus some other piece of information (such as perfmon). ISABPA Report clearly showed the error, which was:
Figure 2 – ISA BPA did it again.
The most interesting part is that since the server was using more and more virtual memory, the amount of paging failure was huge, which means more disk utilization. Since all (ISA, Cache, Logging and Paging File) were in the same disk we started to see disk queue, as result ISA BPA also alerted us that the logging was failing to write in disk:
Figure 3 – Log Write time Excessive warning.
As the log says, if this pattern continues and the time exceeds 30 seconds ISA Server will go into lockdown mode, which in this case it did.
3. Conclusion
This is typical scenario where the server was initially projected to a certain amount load and the planning to change the memory dedicated to cache was not accurately done. As result a complete Internet access downtime and external access to the published services happened. The conclusion is quiet simple: planning is a key element to have a stable server and best experience with the product that you are implementing.
There is nothing better to avoid headaches then a good planning for the technology that you are about to implement. Many scenarios of poor performance or inappropriate behavior sometimes happen because administrators skip the planning phase and go direct to the implementing phase. My advice here is: try to avoid this rush as much as you can, skipping planning phase can cost you more than you think. You just need to imagine the downtime that you might have when the environment is already in production and your server is not capable to handle the amount of requests that you have. Think about it!
From the Forefront Security for Exchange Server perspective we have good news, recently released the Forefront Security for Exchange Server SP1 capacity plan tool. This tool will assist you during the planning phase to correctly size the hardware that is necessary to run this product, among of many other things. Got get it at: http://www.microsoft.com/downloads/details.aspx?FamilyID=522da65d-5263-4f5d-b929-8428a394b9af&displaylang=en
Happy FSE Planning Phase for ya!!
A pal from my hometown this week told me that he just released his book about SQL Server 2008 that is now available at Amazon. This is a great achievement and I’m really proud that you accomplished that. Nice job Herleson and keep up with this energy, enthusiasm and great work that you do in the local community.
Last week I was working on a very interesting case where only one specific user was having a problem browsing Internet through ISA Server 2006. When this specific user was trying to access Internet he receives a page with a time out error message and Firewall client turns red immediately. Interesting part was that regardless of the client workstation where this user was logged in, the behavior always was the same.
The approach used to identify what was going on was:
· Use LDIFDE utility to dump the user account that works and compare to the user account that doesn’t work. You can see more details on how use this utility using http://support.microsoft.com/kb/271201
· Use ISA Data Packager on the ISA Server in repro mode using Web Proxy and Web Publishing template.
· User ISA Data Packager on the client side using Firewall Client template.
Here are some interesting points that were found:
· The user that was having problem was member of hundreds of groups.
· Netmon trace shows KRB5KRB_ERR_RESPONSE_TOO_BIG, which according to http://technet.microsoft.com/en-us/library/bb463166.aspx means “too much data”.
It boils down that the issue was exactly that, the user belongs to many groups and therefore the Kerberos token size was too big. To workaround on that, the following registry key was added to the client workstation:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\Kerberos\MaxTokenSize = 65535
To confirm that you are having this problem you can also use the tool Tokensz and verify how big it is your Kerberos token.
Now it is confirmed, June 19th I will be in Fortaleza/CE/Brazil to deliver a presentation about Forefront TMG, covering some areas such as NIS, Malware Inspection, SMTP Protection and some other news and cool features. It will be good to be back to my hometown and enjoy some time with my friends and family. Here it is the registration page: http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032416163&Culture=pt-BR.
Thanks for the local folks (MVPs and IT Professionals) that are supporting this event:
http://www.mansur.eti.br
http://blog.thiagoeverton.eti.br
http://herleson.spaces.live.com
http://www.jorgebarata.eti.br
...and others :)
See y’all there soon.
This week we launched ISA BPA V7 and this is a great opportunity to continue the explanation on how ISABPA can be useful for proactive and reactive work. Last session I explained how you can use ISA BPA for proactive work with ISA Server. This session will explain the benefits of using ISABPA while troubleshooting an issue.
2. ISA Data Packager
Besides the ISABPA itself, when you install this tool a group of programs is created within Microsoft ISA Server group as you can see in Figure 1:
Figure 1 – Tools that are installed by ISA BPA.
ISA Data Packager is a data gathering tool that can assist you to collect a set of data in one single shot. Let’s use as an example a scenario where user can’t access certain web sites. You can launch the ISA Data Packager and the first screen will present you the templates that are available:
Figure 2 – ISA Templates.
The template that you will choose will depend on the scenario that you are dealing with; here are some examples of usability of the main templates:
Scenario
Template
Unable to access Internet
Web Proxy and Web Publishing
Unable to access some parts of the web site
Get prompt for authentication when accessing a web site
OWA Exchange Publishing rule not working
Sharepoint Publishing rule no working
Unable to establish a VPN Site to Site
VPN
Unable to connect from a client to ISA using PPTP or L2TP
When I open my ISA Console I receive a 0x800 Error and nothing shows up
ISA Administration
ISA Console is crashing
Firewall Policy doesn’t show the rules
Monitoring / Configuration shows one node is not out of Sync
Configuration Storage Server
When accessing Internet Firewall Client turns red.
Firewall Client (NEW in ISABPA7)
Collect data from ISA Server to review the configuration later (no issue to reproduce)
Basic Repro and Static Configuration
Next step is to choose the template according to the scenario, for this example I’m going to chose Web Proxy and Web Publishing. After select and click Next you will see the following screen:
Figure 3 – Summary of the default selections
A set of options are selected by default when you choose the template, those options will vary according to the template that was previously selected. Notice that ISAInfo Report is not selected, which is something that is very useful since with this information you will be able to review all the details from this particular ISA box. In case you want to add that in your data collection you just need to click Modify Options and the following screen will appear:
Figure 4 – Changing default Options.
Here are some other guidelines about this screen:
· If you are having issues such as prompt for authentication when browsing internet, or ISA Server losing the secure channel with the DC, make sure to enable the option Netlogon Logging.
· If you are not dealing with Performance issue, disable the option Performance Monitor Snapshot.
· If you are using MSDE Database for logging and you want to collect data from it, select MSDE Error Logs.
· Change Tracking is NEW in ISABPA7.
After making the selection, click Start Data Collection and wait until the option press space bar to continue appears as shown in Figure 5:
Figure 5 – Starting capture.
At this point you should go to the workstation that is facing the problem and reproduce the issue that you are having. After reproduce the issue press space bar again in the collecting data window and wait until the CAB is generated.
3. Now What?
ISA Data Packager Creates a file called ISAPackage.CAB, by default located in the desktop. This file contains the following folder / files:
Folder
File(s)
Description
BpaDataPackagerLogFiles
BpaDataPackagerLogFile.txt
This is the ISA Data Packager Log that has information about the moment of the data collection. You will use this file to troubleshoot issues where the Data Packager failed to run for example.
IDP.2009-5-7.9-8-29.trace.log
Verbose logging for ISA Data Packager, also used to troubleshoot ISA Data Packager itself.
BpaReportFiles
BPAReport_ISACONTN1_200905070911.xml
This is the ISA BPA Health Check report that you can load using the ISA BPA Tool.
BPAReport_ISACONTN1_200905070911.xml.log
Log for data collection of the ISA BPA, used to troubleshoot ISA Health Check itself.
IsaConfigExport.200905070911.xml
This is the export configuration of the ISA Server. Consider this your backup if you never made one, here it goes.
EventViewerEvents
Application.evt
NEW on BPA7 – this is the export of the Application log in EVT format.
EventViewer_ErrorEvents6.csv
Only errors events (windows event viewer events) in CSV format.
EventViewer_IsaEvents6.csv
Only ISA errors events logged in windows event viewer.
System.evt
NEW on BPA7 – this is the export of the System log in EVT format.
IsaInfoFiles
ISAInfo_isacontn1.log
Log for data collection of the ISA Info, used to troubleshoot ISA Info itself.
ISAInfo_isacontn1.xml
ISAInfo file that you can open using ISA Info tool from ISA Tools .org.
ISALogs
IsaLogs_Firewall_TextEXT_200905070911.csv
Firewall Logging in CSV format.
IsaLogs_WebProxy_TextEXT_200905070911.csv
Web Proxy Logging in CSV format.
IsaTraces
isalog.bin
Files used by Microsoft CSS Engineers only since they require internal symbols to parse it.
manifest.txt
NetworkCaptures
External_20090507090839.cap
Network Capture from the external interface. Correct, you don’t need to start netmon separately when using ISA Data Packager.
LocalCorp_20090507090839.cap
Network Capture from the Internal interface.
Note1: amount of files and folders will vary according to the template that you choose.
Note 2: file name will vary according to the date of the day.
4. Conclusion
With this set of data you have enough data to start troubleshooting the issue that you are facing with ISA Server. You have logs, network captures and capability to read ISA Server configuration. My recommendation is that you install this tool in your lab, and start to test simple scenarios so you get used to read those logs. Try to simulate simple issues in your lab and look the logs to see what you can do to fix the issue.
In a joint effort with MS Brazil (thanks Fabio Hara and Gilson Banin) we are releasing three demos (in Portuguese) about Forefront TMG Beta 2, the videos are:
· Overview of TMG Beta 2 and comparisons between TMG MBE and TMG Beta 2
· Understanding NIS
· SMTP Protection
You can watch the videos at http://technet.microsoft.com/pt-br/dd744741.aspx
Our website has the latest updates on the events that we are going to participate in the next couple of months. Check it Events page at http://www.mstmgbook.org for more information.