We just release a new update for Forefront TMG 2010 called Update 1, this update one includes:
• SafeSearch Enforcement. Forefront TMG can enforce blocking adult text, images and videos from search results by popular search engines. SafeSearch can be enforced on specific groups or to the entire organization. • Including non-primary URL filtering categorizations. Forefront TMG uses an algorithm to select a URL’s “primary” category from among up to four categorizations provided by Microsoft Reputation Services (MRS). In Update 1 you can control access to sites that match any of the non-primary categorizations provided by MRS. For example, a URL with a primary categorization of News can now match a rule by any of its non-primary categorizations (such as Web Mail). • Support for Exchange 2010 SP1 • Bug fixes and various other improvements. Details will be soon available at http://go.microsoft.com/fwlink/?LinkId=201151.
Go get it here.
1. Introduction
Talking about curious case to troubleshoot, this was a very good one. First because this was a new deployment, a well planned hardware for the deployment and the amount of traffic hitting TMG was not that huge. Everything seems to be okay, other than the fact that at least twice a week TMG was stopping responding and it was necessary to restart the whole server. To make things even more bizarre, such issue was happening in random days but always within the range of 8AM and 11AM.
2. Gathering Data
On this scenario TMG administrator couldn’t really get a dump of the wspsrv.exe process because he couldn’t even log in locally. It was necessary to prepare the machine for a kernel dump capture, for that I used the procedures from KB969028.
3. Analyzing the Data
On a scenario where the machine completely stops responding one of the most useful commands to type while analyze a kernel dump is the !locks command. By definition (windbg help), the !locks command will:
“Display all locks held on resources by threads. A lock can be shared or exclusive, which means no other threads can gain access to that resource. This information is useful when a deadlock occurs on a system. A deadlock is caused by one non-executing thread holding an exclusive lock on a resource that the executing thread needs.”
In this case here it is the result of this command:
0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks...........................................................................................................................................
Resource @ 0xfffffa800dbad748 Shared 1 owning threads
Contention Count = 493
Threads: fffffa800e09db63-01<*> *** Actual Thread fffffa800e09db60
KD: Scanning for held locks......
Resource @ 0xfffffa800bc2a7b8 Exclusively owned
Threads: fffffa800e09db60-01<*>
KD: Scanning for held locks..........................................................................................
7461 total locks, 2 locks currently held
Next step is to verify which thread is that:
0: kd> !thread fffffa800e09db60
THREAD fffffa800e09db60 Cid 0004.0680 Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (UserRequest) KernelMode Non-Alertable
fffffa800cfc4060 SynchronizationEvent
IRP List:
fffffa8010231bd0: (0006,0430) Flags: 00060901 Mdl: fffffa8010287000
Not impersonating
DeviceMap fffff8a000008b30
Owning Process fffffa8009a3a040 Image: System
Attached Process N/A Image: N/A
Wait Start TickCount 26737768 Ticks: 3 (0:00:00:00.046)
Context Switch Count 940461
UserTime 00:00:00.000
KernelTime 00:13:42.562
Win32 Start Address vhdmp!VhdmpiAsyncOpThread (0xfffff880076bcdb0)
Stack Init fffff88008267db0 Current fffff88008267440
Base fffff88008268000 Limit fffff88008262000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP RetAddr : Args to Child : Call Site
fffff880`08267480 fffff800`01ac9992 : 00000000`0000000c fffffa80`0e09db60 fffffa80`00000000 fffffa80`10231bd0 : nt!KiSwapContext+0x7a
fffff880`082675c0 fffff800`01acbcff : 00000000`0000139c fffffa80`0cfc4060 00000000`00000000 fffffa80`0bf1a9a0 : nt!KiCommitThreadWait+0x1d2
fffff880`08267650 fffff800`01dbd1d2 : 00000000`00000000 ffffffff`00000006 00000000`00000000 00000000`00000000 : nt!KeWaitForSingleObject+0x19f
fffff880`082676f0 fffff800`01ac1993 : fffffa80`0e09db60 00000000`00200200 00000000`00000000 fffffa80`0cfc4060 : nt!NtWaitForSingleObject+0xb2
fffff880`08267760 fffff800`01abdf30 : fffff880`07694640 fffffa80`0cde80e0 fffffa80`0bc2a000 ffffffff`8000139c : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`08267760)
fffff880`082678f8 fffff880`07694640 : fffffa80`0cde80e0 fffffa80`0bc2a000 ffffffff`8000139c 00000000`000001d1 : nt!KiServiceLinkage
fffff880`08267900 fffff880`076b7ee2 : 00000000`00000000 fffff880`076b7e5c fffffa80`0cde80e0 fffffa80`0cde80e0 : vhdmp!VhdmpiReadVhdFileAtPassiveLevel+0x50
fffff880`08267970 fffff880`076bd9e0 : 00000000`00000000 00000000`000001d1 fffff8a0`0c2f2000 00000000`00001d10 : vhdmp!VhdmpiCopyRawSectorsSync+0x72
fffff880`082679c0 fffff880`076bdf11 : fffff8a0`01f26000 00000000`00022dbe fffff880`08267b90 fffff880`08267ac8 : vhdmp!VhdmpiRegisterWithRootEnumerator+0x220
fffff880`08267a60 fffff880`076c05e4 : 00000000`00000001 00000000`00000000 00000000`00022dbe fffffa80`0bc2a000 : vhdmp!VhdmpiShutdownNestingLevels+0x3b1
fffff880`08267b60 fffff880`076c083a : 00000000`00000000 00000000`00000001 fffffa80`0e573c00 fffffa80`0e573c00 : vhdmp!VhdmpiDoCompaction+0x124
fffff880`08267ca0 fffff880`076bce02 : ffffffff`ff676980 fffffa80`0e09db60 00000000`00000080 fffffa80`0cde80e0 : vhdmp!VhdmpiCompactThread+0x13a
fffff880`08267d00 fffff800`01d66c06 : fffffa80`0e09db60 00000000`00000080 fffffa80`09a3a040 00000000`00000001 : vhdmp!VhdmpiAsyncOpThread+0x52
fffff880`08267d40 fffff800`01aa0c26 : fffff880`009bf180 fffffa80`0e09db60 fffffa80`09a3ab60 fffff880`01452534 : nt!PspSystemThreadStartup+0x5a
fffff880`08267d80 00000000`00000000 : fffff880`08268000 fffff880`08262000 fffff880`08267450 00000000`00000000 : nt!KxStartSystemThread+0x16
Notice that this thread has an I/O request packet (IRP) and for scenarios like this (server freezes) it is very important to review the IRP for that thread that seems to be causing the issue. To do that you need to use the command !irp, here it is the result for this case:
0: kd> !irp fffffa8010231bd0
Irp is active with 12 stacks 9 is current (= 0xfffffa8010231ee0)
Mdl=fffffa8010287000: No System Buffer: Thread fffffa800e09db60: Irp stack trace.
cmd flg cl Device File Completion-Context
[ 0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
>[ 3,34] 10 e1 fffffa800a03f060 00000000 fffff88000e34180-fffffa800a04de20 Success Error Cancel pending
\Driver\Disk volmgr!VmpReadWriteCompletionRoutine
Args: 00200200 00000000 3bd4e26200 00000000
[ 3, 0] 0 e0 fffffa800a04dcd0 00000000 fffff88001792ee0-fffffa800de75450 Success Error Cancel
\Driver\volmgr volsnap!VspReadCompletionRoutine
Args: 00200200 00000000 3bd4d26200 00000000
[ 3, 0] 0 e1 fffffa800a047040 00000000 fffff8800145ef70-fffffa800dabd550 Success Error Cancel pending
\Driver\volsnap Ntfs!NtfsMasterIrpAsyncCompletionRoutine
[ 3, 0] 0 1 fffffa800a069030 fffffa800df75880 00000000-00000000 pending
\FileSystem\Ntfs
Args: 00200200 00000000 21ed477200 00000000
The major function 3,34 has the pending status set, which appears to be stuck on \Drover\Disk (Volume Managr – volmgr). Let’s take a closer look on the value fffffa800a03f060, which we can see by using the command !devobj as shown below:
0: kd> !devobj fffffa800a03f060
Device object (fffffa800a03f060) is for:
DR2 \Driver\Disk DriverObject fffffa8009e53e70
Current Irp 00000000 RefCount 0 Type 00000007 Flags 01002050
Vpb fffffa8009fd50b0 Dacl fffff9a10043d7f0 DevExt fffffa800a03f1b0 DevObjExt fffffa800a03f858 Dope fffffa8009feb450
ExtensionFlags (0x00000800)
Unknown flags 0x00000800
AttachedDevice (Upper) fffffa800a03fb90 \Driver\partmgr
AttachedTo (Lower) fffffa8009fe9b70 \Driver\USBSTOR
This is a USB driver, which is something that I was not expecting to have a this TMG box at all, so I decided to before even move forward ask to the TMG administrator what a USB driver is doing on TMG. TMG Administrator then clarified the scenario as follow: “We have a USB external driver attached to the system to store the full backup of the system that we perform every night.”
The dump is quiet clear, in other words: server is waiting for the disk. At this point I can conclude that the server was hanging waiting for this USB driver, which for some reason was not responding, as a consequence the whole box stop responding (including TMG, but clearly not a TMG issue).
4. Conclusion
After start reviewing the logs from the backup software we could conclude that: the backup job was supposed to finish at 6AM, however some days was extending beyond that, when the backup was running ahead of 8AM and the production started on TMG the system presented this behavior. Backup software was performing a full backup of the system, including TMG cache folder (which by itself was 4GB). We do not recommend making a backup of TMG cache folder, this folder needs to be excluded from backup. Making backup of the cache folder can cause other issue of this nature, such as crashing Firewall Service as I mentioned in this post. To resolve this problem we excluded the TMG cache folder from the backup and re-schedule the backup job to start at 11PM and let it run until 7AM in an attempt to avoid backup job to go beyond commercial business hours.
I would like to take some time today to bring to y’all attention some new books that are out there on Forefront UAG and TMG technologies. My buddies Ben Ari and Ran Dolev (both from CSS UAG Team) are finishing writing the new Microsoft Forefront UAG 2010 Administrator's Handbook, since those guys are very sharp on UAG you surely can expect a great content. Another new book out there about TMG is from the Forefront MVP Tamás Gál, more info about the book (in Hungarian) is available here. Now that me and Tomas pretty much finished writing the Forefront books series (see previous blog post) I can take some time to finish another project that I’m working since December last year, which is the Security+ Certification Guide book (in Portuguese only). This one I should be done writing by next month (October) and the book will be released in Brazil by April 2011, more info will be available at www.securityplusbr.org.
Just a quick update on the Forefront eBooks,
Me and Tom just finished writing and reviewing all three books, today we wrote the About the Authors and Acknowledgements. The content is now on the final stage with MSPress and the books are available for pre-order at Amazon.com, check it out at:
Enjoy it !!
Recently I assisted an IT Pro on the Forum thread http://social.technet.microsoft.com/Forums/en-US/Forefrontedgesetup/thread/c7606ead-5957-4ef8-a4e9-e5aa85493581 where the problem was related to this behavior described in the title of this post. The problem happens very precisely, after applying TMG SP1 the TMG Managed Control Service takes too long to start, although it does start. As I describe in the Forum, if you are facing this issue and wants to use the private hotfix for that you will need to open a support case with Microsoft. If the hotfix resolves the problem the case won’t be charged, however if the hotfix does not fix the issue this means that the problem is not the same although the symptom could be similar. In other words, if further troubleshooting is necessary the case will be normally charged. To open a support case use this link http://support.microsoft.com/select/Default.aspx?target=assistance , select Forefront Threat Management Gateway (under security) and follow the wizard.
Whatta busy week here with this problem. As you might be aware, Exchange 2010 SP1 breaks the integration that TMG has with Exchange Edge when using E-Mail Protection feature, the issue causes the TMG Managed Control Service to fail to start. This issue is documented on our team blog http://blogs.technet.com/b/isablog/archive/2010/09/01/problems-when-installing-exchange-2010-service-pack-1-on-a-tmg-configured-for-mail-protection.aspx and on Exchange Team blog at http://msexchangeteam.com/archive/2010/09/01/456094.aspx. We are working on a hotfix for this issue and more news will be posted on the TMG Team Blog as soon as we test the fix in our lab and we confirm that works.
One thing that it is important to keep in mind is the following: don’t make changes directly on Exchange or FPE installed on the TMG Server when this issue happens. What I mean with that? Here it is the issue flow so you can better understand:
The reason behind that is because if you make changes, there is a possibility that once we fix the original issue (with the hotfix that will come up soon), TMG Managed Control Service will still failing to start with the error message below:
…and on Event Viewer you might see the error below:
Log Name: Application
Source: Microsoft Forefront TMG Control
Event ID: 31308
Task Category: None
Level: Error
Keywords: Classic
Description:
The Forefront TMG Managed Control service failed to initialize. Error information: Value does not fall within the expected range.
This error has nothing to do with the issue introduced by Exchange 2010 SP1, this is another problem that can be caused by changes that are made directly on those products and when TMG Managed Control Service tries to synchronize the config, it fails. It is important to mention that this failure doesn’t always happen, the normal behavior is to TMG detect the changes and undo it. When TMG is able to do that you will see the following alert:
Solution? Undo the manual changes that were done via those products' console and make sure to always use TMG to manage the configuration among those products (at least this is the behavior now TMG RTM and SP1). In this particular case that TMG Managed Control Service is down, you can’t make changes via TMG console because it will not sync with Exchange Edge and FEP, therefore you should hold any change to be done only after the hotfix for this issue (Exchange 2010 SP1 problem) is released and you had applied on the system.
Nowadays information is much easier to find than it was in the past, you can find information about pretty much everything by just searching on the web (try http://www.letmebingthatforyou.com/?q=dns%20atacck for example). For this reason it is even more important to be diligent while protecting your company’s data by avoiding data leakage and data enumeration.
One of the pre-attack phases is exactly the enumeration of data for the target system (or company), during this phase the attacker will try to gather as much information as possible about the target, going from the basic stuff that it can find about the target on the company’s web site to a more detail data enumeration using tools and techniques. One of the steps for data enumeration is query the target DNS system in order to obtain more information about hosts and IPs. This process can start by simply using the nslookup command. Assuming a successfully query, the attacker can try to transfer all the information for that domain by attempting a zone transfer operation. This post will explain how to configure your DNS Publishing rule on TMG to avoid zone transfer.
2. What you want to avoid?
The scenario that I described above is an unfortunate common scenario, many times because the system administrator forgot to securely configure the DNS Server on the internal network to not allow zone transfer for all systems. Here an example of a DNS that is wide open for zone transfer:
When this configuration is place the following result will happen if the client tries to transfer a zone using nslookup:
Microsoft Windows [Version 6.0.6002] Copyright (c) 2006 Microsoft Corporation. All rights reserved.
C:\>nslookup Default Server: mysrv Address: 192.168.1.10 > server 192.168.1.154 Default Server: TMGFW Address: 192.168.1.154
> set q=all
> contoso.com
Server: TMGFW Address: 192.168.1.154 contoso.com internet address = 10.20.20.10 contoso.com nameserver = dc01.contoso.com contoso.com primary name server = dc01.contoso.com responsible mail addr = hostmaster.contoso.com serial = 494 refresh = 900 (15 mins) retry = 600 (10 mins) expire = 86400 (1 day) default TTL = 3600 (1 hour) dc01.contoso.com internet address = 10.20.20.10
> ls -d contoso.com
[TMGFW.home] contoso.com. SOA dc01.contoso.com hostmaster.contoso.com. (494 900 600 86400 3600) contoso.com. A 10.20.20.10 contoso.com. NS dc01.contoso.com _msdcs NS dc01.contoso.com _gc._tcp.Default-First-Site-Name._sites SRV priority=0, weight=100, port=3268, dc01.contoso.com _kerberos._tcp.Default-First-Site-Name._sites SRV priority=0, weight=100, port=88, dc01.contoso.com _ldap._tcp.Default-First-Site-Name._sites SRV priority=0, weight=100, port=389, dc01.contoso.com _gc._tcp SRV priority=0, weight=100, port=3268, dc01.contoso.com _kerberos._tcp SRV priority=0, weight=100, port=88, dc01.contoso.com _kpasswd._tcp SRV priority=0, weight=100, port=464, dc01.contoso.com _ldap._tcp SRV priority=0, weight=100, port=389, dc01.contoso.com _kerberos._udp SRV priority=0, weight=100, port=88, dc01.contoso.com _kpasswd._udp SRV priority=0, weight=100, port=464, dc01.contoso.com casrv A 10.20.20.9 dc01 A 10.20.20.10 DomainDnsZones A 10.20.20.10 _ldap._tcp.Default-First-Site-Name._sites.DomainDnsZones SRV priority=0, weight=100, port=389, dc01.contoso.com _ldap._tcp.DomainDnsZones SRV priority=0, weight=100, port=389, dc01.contoso.com EXSRV A 10.20.20.11 farm CNAME exsrv.contoso.com ForestDnsZones A 10.20.20.10 _ldap._tcp.Default-First-Site-Name._sites.ForestDnsZones SRV priority=0, weight=100, port=389, dc01.contoso.com _ldap._tcp.ForestDnsZones SRV priority=0, weight=100, port=389, dc01.contoso.com TMGFW A 10.20.20.1 TMGSTD A 10.20.20.31 UAG10 A 10.20.20.12 UAG11 A 10.20.20.13 WKSVista A 10.20.20.152 contoso.com. SOA dc01.contoso.com hostmaster.contoso.com. (494 900 600 86400 3600)
Well, this is not nice at all…here it is my whole internal domain zone exposed to all Internet users just because the system administrator forgot to disable one checkbox.
3. Protecting Internal Resources
If system administrator didn’t do his homework, you as firewall admin MUST do it ASAP. However it is very important to emphasize that this band aid solution on the perimeter shouldn’t be the main one, we always need to fix the root cause of the problem, which in this case is to securely configure the DNS Server. The fact that you are protecting the DNS Publishing on your perimeter is just a countermeasure to avoid a potential attempt to transfer the internal DNS zone to an external (non trusted) resource.
The DNS Filter on TMG is enabled by default as shown below:
This filter has the following capabilities pre-configured by default:
As zone transfer can or cannot be a legitimate type of access the parameter DNS_Zone_Transfer is set to 0 by default. In other words: if you want to allow zone transfer you should leave this parameter as is. The recommendation is to make sure to allow zone transfer only to trusted DNS servers, which means that you should configure your DNZ zone to allow transfer only for the server’s IPs that you trust. If all the servers that you trust are behind TMG (on the internal network) then you can safely block zone transfer attempts from traffic coming from the external (Internet) network. To do that you need to run the following script:
Set root = CreateObject("FPC.Root") Set isaArray = root.GetContainingArray() Set attackDetection = isaArray.ArrayPolicy.AttackDetection Set dnsFilter = isaArray.Extensions.ApplicationFilters.Item("{49FE2B2F-3BB4-495C-87C8-3890C3C35756}")
dnsFilter.Enabled = True Set vpSets = dnsFilter.VendorParametersSets
On Error Resume Next vpSets.Add "{D96C5E7F-5B13-4E1A-94A1-36CA7B54604E}", False, False On Error Goto 0 ' If a vendor parameters sets for DNS Filter already exists
vpSets.Item("{D96C5E7F-5B13-4E1A-94A1-36CA7B54604E}").Value("DNS_Zone_Transfer") = "1"
vpSets.Save
…OR you can also use the UI to make this change using the option below:
Note: if you run the script while the console is open and you try to see if this option is enabled on the UI, the option might still be disabled. To refresh it, close TMG console and open it again.
After running this script (or changing it on the UI), make sure to restart Firewall Service and then try again to run the nslookup ls command from an external computer. The expected result is shown below:
[TMGFW]
*** Can't list domain contoso.com: Unspecified error
The DNS server refused to transfer the zone contoso.com to your computer. If this is incorrect, check the zone transfer security settings for contoso.com on the DNS server at IP address 192.168.1.154.
Now this for sure is a better result for an external user to get it.
TMG installation problems can be a bit trick to troubleshooting if you don’t know which components are involved, however if you know then things start make more sense. Most of the setup problems that I faced up to now on TMG 2010 (since RTM) were related to ADLDS or some kind of domain connectivity problem. The most two recent examples are described on two articles that I tech reviewed from my friends Bala Natarajan and Niladri Dasgupta wrote:
Last week I worked on an issue where TMG admin was not able to install this brand new TMG to be used as Edge Firewall. The error message that he was receiving was:
After this error the setup process rolled back and finish without completing the installation. As recommended on the previous two articles mentioned in this post, the first step is to review the setup logs and look for more information in order to move the troubleshooting further. In the ADAM Log file we can see the following entry:
When you see an error where trust relationship between client and domain is failing, be sure to do your homework, in other words, check:
When I hit the third test I found out the problem:
This was the problem, because Windows (where I was trying to install TMG) was sending the traffic to the wrong interface. Once we moved the Internal to the top, flushdns (with ipconfig) and ran the setup again the issue went away and the installation finished successfully.
Note: same recommendation to have Internal on the top applies to UAG, check it out a great reference on that written by Jason Jones at http://blog.msedge.org.uk/2010/04/recommended-network-card-configuration_14.html
Slow browsing experience is a behavior that can happen for so many reasons that cover everything in one single article is just not feasible, mainly when the list still growing. Here are some posts/articles that I wrote on this matter:
The list is growing because recently, troubleshooting another issue of this nature today I learned another cool thing: disabled adapters matters. What I mean with this is that if you have multiple adapters on your ISA/TMG and if some of the adapters are not in use and are disabled, this can still affect the performance of the system. How? According to the binding order. Remember that I talked about binding order here some time ago? The DNS best practice analyzer pointed me to the right direction on this, here what he said about disabled adapters on the top of the binding order:
Issue
Valid network interfaces should precede invalid interfaces in the binding order. A disabled or invalid adapter precedes a valid adapter in the network interface binding order list.
Impact
The binding order determines when network interfaces will be used to make network connections by the computer. A disabled adapter high in the binding order can degrade performance.
Resolution
Move all disabled and invalid interfaces to the bottom of the binding order list.
Keep this caveat in mind during your ISA/TMG performance health check analyzes.
Introduction
It could be just another case where we grab some data, look at it and resolve the issue, but it was a long road until we get to the bottom of this. The scenario was quiet simple: clients that were behind TMG couldn’t access HTTPS sites. All other sites using HTTP were working just fine and the most phantasmagoric thing that we noticed was that on TMG Logging we just have entries for 8080, no HTTPS request at all as show below:
On the client, the only thing that was showing up was:
Troubleshooting
All the usual suspects during a standard troubleshooting phase were eliminated (HTTPS Inspection, Malware Inspection, NIS, URL Filtering, HTTP Filter, Network, etc), it was time to thing: what else is left? What about the client? Let’s take a netmon on the client and see what’s going, right? Yep…let’s do it. For my surprise, after starting netmon and repro the issue, we did not even saw the HTTPS Request leaving the workstation; we just saw requests on port 8080 and then the clients sent a FIN:
Digging Deeper
In order to investigate further what was going on I used the tool TCPView, which gives more details information about network connectivity and process that are using for a specific connection. So I tried to access the Bank of America site (which uses HTTPS by default) and here it is the result:
Notice two things:
Note1: I’m not saying the driver name because this behavior was observed in many others third party AV filter drivers.
Ok, now we have some footprints that we can use to investigate further. The process XXX.exe belongs to a third party antivirus. As a temporary test we disabled the AV solution using MSConfig and restarted the workstation. It didn’t help since there were some other kernel drivers loaded in the system. As a temporary measure we uninstalled the AV, tested and worked. At that point we handled over the issue to the AV Vendor.
Note2: Only disable AV solution on the workstation for troubleshooting purpose, validate your tests and enabled it back. It is strongly recommended to have antvirus solution on your system and address potential issues with the AV vendor.
Just another reminder that not even everything that you think is caused by TMG, it actually is….this is just another example of TMG (as it was on ISA) be a victim of another problem.
Wow, 5 months already passed since the TMG book was released, looks like it was yesterday that me Mohit and Jim started this project back in March 2008. Well, time past really fast….anyway, this is a quick post to talk about our Forefront TMG Book and the great feedback that we are receiving so far from TMG Book readers around the world (more than two thousand already have this book). Besides the fact that we received very good feedbacks directly from readers, there are also some good feedbacks on sites such as Amazon and also great reviews from some folks at Microsoft, such as Alan and independent reviewers such as New Signature Creative Technology and in particular Left Brain bookstore that says:
The book is available at:
Last but not least I would like to say thank you for all of you that acquire this book and support this product.
Were you at TechEd US 2010? If you were I hope that you saw the presentation below. If you were not, then here it is a good chance to watch it and learn more about TMG.
PPT Slides are located at http://ecn.channel9.msdn.com/o9/te/NorthAmerica/2010/pptx/SIA308.pptx
Video is available for download at http://ecn.channel9.msdn.com/o9/te/NorthAmerica/2010/wmv/SIA308.wmv
Here it is another set of slides, now about TMG from a presentation that I delivered last year in Brazil.
Once again, enjoy it.
Today a friend of mine asked where he can get the slides that once were available at http://www.microsoft.com/brasil/technet/eventos/ciclocontinuado/ciclo-nlb.mspx, the slides are not there anymore, so I decided to share it here:
Enjoy it.
We just released a new KB where we explain more details about Malware Inspection and NIS files. Check it out at http://support.microsoft.com/kb/2160835.
Today I was assisting a friend of mine here from TMG team that was facing this issue, same issue that was also mentioned on this thread. The problem was happening when using Cryptography Next Generation (CNG) or also called V3, TMG was not recognizing the private key and was showing up this error message. This is a known issue because TMG (and ISA) don’t support CNG (V3 Certificates). This is well documented under the unsupported documentation here:
Issue: Forefront TMG does not support the use of certificates created using CNG (Certificate New Generation) based templates for Web listeners or as client certificate authentication in Web publishing or Web chaining rules.
Cause: CNG certificates are not usable by Forefront TMG.
Workaround: Create certificates using Windows 2000 or Windows 2003 templates.
From: http://technet.microsoft.com/en-us/library/ee796231.aspx#dfg9o9i8uuy6tre
Again, make sure to read this unsupported document before deploy TMG, there you will find the official statement from TMG Product Team about what it is supposed to work and what it is not.
Note: Important to emphasize that CNG V3 is not X.509 V3. CNG V3 refers to the new V3 Certificate Template on 2008 while X.509 V3 is the current certificate standard in which TMG is fully compatible.
Recently I was engaged on this TechNet Forum thread and I thought that it was an interesting question and concern. Let me explain the scenario on which I was able to repro the behavior on my own lab:
Contoso has a security policy where only users that belong to the Internet Users group can access Internet. To enforce that they created a rule on ISA Server on which only members of the Internet Group can access Internet. Contoso uses Internet Explorer 6 on all workstations of their network. The process today is that once the user receives authorization to access Internet, the AD Admin adds the user to Internet Users group and the user just need to open a new browser session and he will have access. Recently Contoso upgraded 50% of their workstations to use Internet Explorer 8, they notice the following behavior since this change: if an user that doesn’t belong to Internet Users group is already logged on the workstation and the administrator add this user to the Internet User’s group, the user receives an error when try to browse Internet, the error says that ISA denied the request. If the user logs off from the windows and logon again it works fine.
Why this is happening?
What it is happening on this case is simply the way that Internet Explorer 8 works for the authentication part of the conversation. Since the version 7, Internet Explorer was capable to use Kerberos for proxy authentication (while IE6 and lower only use NTLM and Basic). ISA Server will negotiate the authentication with the browser and will authenticate according to the method that is supported on both sides. The difference in this case is the following:
With IE7 (and higher)
1. Client sends the GET request to www.microsoft.com (for example). This request goes as anonymous.
2. ISA will send the 407 asking for authentication.
3. If client has already a Kerberos cached ticket it will not go to the DC to get a new one, if there is no ticket then it will go to the DC to get an updated ticket to send to ISA.
4. Client will send another GET Request now with the credentials and the ticket.
5. ISA will verify the request and allow (or deny according to the rule).
Note: in this case since ISA doesn't go to the DC, it relies on user's token to access the resource. Since the current ticket doesn’t include the update on user’s group membership, ISA will verify that the user doesn't below to the group and will deny the request.
With IE6
3. Client sends another GET request with the credentials (NTLM).
4. ISA goes to the DC to authenticate the user.
Note: since ISA goes to the DC it will get an updated version of the user's group membership and will verify that the user now belongs to this group.
5. ISA will allow the user to pass through.
Under the Hood
Prior to get to this conclusion I decided to do some tests to validate the theory and the tests worked pretty fine. Here it is the traffic from the client workstation with the user “Yuri” logged in, right after the inclusion of the user to Internet User’s group:
1. Client sends the GET Request
10.20.20.201 10.20.20.1 HTTP HTTP:Request, GET http://www.microsoft.com/
2. ISA asks for authentication:
10.20.20.1 10.20.20.201 HTTP HTTP:Response, HTTP/1.1, Status Code = 407, URL: http://www.microsoft.com/
Connection: Authentication
ProtocolVersion: HTTP/1.1
StatusCode: 407, Proxy authentication required
Reason: Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy filter is denied. )
Via: 1.1 ISACONTN1
+ ProxyAuthenticate: Negotiate
Proxy-Authenticate:
+ ProxyAuthenticate: Kerberos
+ ProxyAuthenticate: NTLM
Connection:
Connection: Keep-Alive
ProxyConnection: Keep-Alive
Pragma: no-cache
Cache-Control: no-cache
+ ContentType: text/html
ContentLength: 4113
HeaderEnd: CRLF
+ payload: HttpContentType = text/html
Note: Notice that ISA is negotiating the authentication method.
3. Client sends another GET request now with the credential.
10.20.20.201 10.20.20.1 HTTP HTTP:Request, GET http://www.microsoft.com/ , Using SPNEGO Authorization
Command: GET
+ URI: http://www.microsoft.com/
Accept: */*
Accept-Language: en-us
UserAgent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322)
Accept-Encoding: gzip, deflate
Cookie: MUID=42623805587D4F6EB894604864D99E47; WT_FPC=id=173.74.31.197-401582496.30090674:lv=1279361835221:ss=1279361835221; MC1=GUID=ba325a1581e1aa4dbcd0988320053feb&HASH=155a&LV=20107&V=3; A=I&I=AxUFAAAAAABDBgAAgb+LoGxG8UwjOO2L0QId7Q!!
- ProxyAuthorization: Negotiate
- Authorization: Negotiate YIIE5QYGKwYBBQUCoIIE2TCCBNWgJDAiBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKwYBBAGCNwICCqKCBKsEggSnYIIEowYJKoZIhvcSAQICAQBuggSSMIIEjqADAgEFoQMCAQ6iBwMFACAAAACjggO7YYIDtzCCA7OgAwIBBaEOGwxDT05UT1NPLk1TRlSiKTAnoAMCAQKhIDAeGwRIVFRQGxZpc2Fjb250bjEuY
WhiteSpace:
- NegotiateAuthorization:
Scheme: Negotiate
- GssapiKrb5: 0x1
Kerberos:
Note: Notice that client is using Kerberos.
4. This ticket doesn’t contain an updated information about user’s group membership, hence ISA Server sends a deny.
10.20.20.1 10.20.20.201 HTTP HTTP:Response, HTTP/1.1, Status Code = 502, URL: http://www.microsoft.com/
StatusCode: 502, Bad gateway
Reason: Proxy Error ( The ISA Server denied the specified Uniform Resource Locator (URL). )
Connection: close
ProxyConnection: close
ContentLength: 4059
5. At this point, I ran the command klist on the workstation and here are the tickets that I had at that point:
C:\Program Files\Windows Resource Kits\Tools>klist tickets
Cached Tickets: (3)
Server: krbtgt/CONTOSO.MSFT@CONTOSO.MSFT
KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)
End Time: 7/17/2010 18:34:46
Renew Time: 7/24/2010 8:34:46
Server: HTTP/isacontn1.contoso.msft@CONTOSO.MSFT
Server: host/client1.contoso.msft@CONTOSO.MSFT
6. I thought to myself: well, if I wipe all the tickets I will force the user to go to the DC and get an updated ticket, this way if I try again to access Internet, I will use the updated ticket. To do that I ran the command: klist purge. I deleted all tickets and the end result was:
C:\Program Files\Windows Resource Kits\Tools>klist purge
Cached Tickets: (0)
Now is time for the truth, tried to access Internet again and then it WORKED. Here it is the netmon trace for this second attempt with my Kerberos ticket cache clean:
3. Client goes to the DC to get ticket for the user “Yuri”:
10.20.20.201 10.20.20.20 KerberosV5 KerberosV5:AS Request Cname: yuri Realm: CONTOSO.MSFT Sname: krbtgt/CONTOSO.MSFT
- Kerberos: AS Request Cname: yuri Realm: CONTOSO.MSFT Sname: krbtgt/CONTOSO.MSFT
- AsReq: Kerberos AS Request
+ ApplicationTag:
+ KdcReq: KRB_AS_REQ (10)
4. The DC responds:
10.20.20.20 10.20.20.201 KerberosV5 KerberosV5:AS Response Ticket[Realm: CONTOSO.MSFT, Sname: krbtgt/CONTOSO.MSFT]
- Kerberos: AS Response Ticket[Realm: CONTOSO.MSFT, Sname: krbtgt/CONTOSO.MSFT]
- AsRep: Kerberos AS Response
+ KdcRep: KRB_AS_REP (11)
5. Client goes again to the DC to get a ticket to the HTTP Service (ISA SPN):
10.20.20.201 10.20.20.20 KerberosV5 KerberosV5:TGS Request Realm: CONTOSO.MSFT Sname: HTTP/isacontn1.contoso.msft
- Kerberos: TGS Request Realm: CONTOSO.MSFT Sname: HTTP/isacontn1.contoso.msft
- TgsReq: Kerberos TGS Request
+ KdcReq: KRB_TGS_REQ (12)
6. The DC responds:
10.20.20.20 10.20.20.201 KerberosV5 KerberosV5:TGS Response Cname: Yuri
- Kerberos: TGS Response Cname: Yuri
- TgsRep: Kerberos TGS Response
- KdcRep: KRB_TGS_REP (13)
+ SequenceHeader:
+ Tag0:
+ PvNo: 5
+ Tag1:
+ MsgType: KRB_TGS_REP (13)
+ Tag3:
+ Crealm: CONTOSO.MSFT
+ Tag4:
+ Cname: Yuri
+ Tag5:
+ Ticket: Realm: CONTOSO.MSFT, Sname: HTTP/isacontn1.contoso.msft
+ Tag6:
+ EncPart:
7. Client sends the GET request to ISA now with the new ticket:
Host: www.microsoft.com
- Authorization: Negotiate YIIFDQYGKwYBBQUCoIIFATCCBP2gJDAiBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKwYBBAGCNwICCqKCBNMEggTPYIIEywYJKoZIhvcSAQICAQBuggS6MIIEtqADAgEFoQMCAQ6iBwMFACAAAACjggPjYYID3zCCA9ugAwIBBaEOGwxDT05UT1NPLk1TRlSiKTAnoAMCAQKhIDAeGwRIVFRQGxZpc2Fjb250bjEuY
8. ISA allows the traffic:
10.20.20.1 10.20.20.201 HTTP HTTP:Response, HTTP/1.1, Status Code = 302, URL: http://www.microsoft.com/
Conclusion
As you could see on this post this is not really an issue, neither on ISA nor on Internet Explorer, this is actually a behavioral change that Internet Explorer 7 and higher had. This is a good improvement, because Kerberos is much lighter and doesn’t put too much pressure on the DC. One classic problem with NTLM is that the authentication request against the DC is huge, read the article Improving Web Proxy Client Authentication Performance on ISA Server 2006 that I wrote couple of years ago and you will see the advantage that IE7 has over IE6.
But, if it is a requirement for your business that changes to group membership immediately reflect on user’s Internet browsing experience than you can use ISA’s IP address on the IE setting rather than the name, this way you are forcing to use NTLM rather than Kerberos. Just keep in mind that when too much pressure is added on the DC you can face the infamous “random prompt for authentication” on ISA due the 5783 and 5719 events.
Vacation is always good, isn’t it? I was on vacation in June and couldn’t write too much but now I’m back and fully charged for another TMG semester. Last week I was delivering a TMG workshop for Microsoft Premier Customers and some interesting questions appeared during those three days. Among all those questions there were two that I would like to share it here:
1) Can I use ISP-Redundancy to send HTTP through one link and SMTP through another?
No, you can’t. This is unsupported and documented here.
2) Can I install TMG Firewall on EMS?
No, you can’t. This is also part of the unsupported document that you can find here.
It is very important to read this unsupported document before make decisions on your environment. Matter of fact one of the slides that I presented during this workshop last week says:
If you want to be on the safe side never skip phases and during the planning phase make sure to read the unsupported document to see if there is any scenario that might affect your supportability state.
Have a great TMG deployment.
As we just announced on Microsoft Press blog, three new Forefront eBooks are coming soon, check it out the full blog MS Press Blog. Here are they:
We hope you enjoy it.
Cheers !!
This is not only an update for TMG 2010, this is a great enhancement on some of the features available on TMG 2010. Download TMG 2010 SP1 from:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=f0fd5770-7360-4916-a5be-a88a0fd76c7c
…and make sure to take a look at Richard’s article about SP1 new features at:
http://www.isaserver.org/tutorials/Whats-New-Forefront-Threat-Management-Gateway-TMG-2010-Service-Pack1.html
Cheers and enjoy SP1 !!
Today I spoke for two audiences about Microsoft Business Ready Security strategy and TMG. The first presentation was sponsored by SECRELNET, one of the biggest hosting and Internet companies in Fortaleza, this presentation was target to IT Managers and Security Analysts. I would like to say thanks for your active participation – OBRIGADO.
After finishing up the presentation there I went to SEBRAE-CE to speak about TMG 2010 in a SWG (Secure Web Gateway) scenario, the presentation was great. I also had a chance to see some old friends and sign some TMG Books. I had an amazing time, thanks a lot for your participation on this event and also thanks to: Coresec, SECRELNET and SEBRAE for sponsoring this event in partnership with Microsoft.
Inscrições: https://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032453033&Culture=pt-BR
Did you know that now Microsoft CSS (Customer Service and Support) has a new offer for customers without a formal contract with Microsoft? We do, it is called Advisory Services and for the ISA/TMG space we have some great offers that can help you during a deployment or migration phase. Check it out our current offers:
Microsoft Advisory Services Engagement Scenario - New ISA/TMG Deployment & Configuration
Microsoft Advisory Services Engagement Scenario - ISA to TMG Migration
Microsoft Advisory Services Engagement Scenario - ISA/TMG Server Virtualization
Microsoft Advisory Services Engagement Scenario - ISA/TMG branch office setup
For those that loves TMG and ISA BPA, now you can enjoy the same experience on the UAG world, yesterday we released the UAG BPA, with this tool you can get details on critical configuration issues, potential problems, and information about the local UAG computer. UAG BPA is now available at:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=d24994ef-8670-4324-957a-805d35f1244e
Many companies are starting to budget for the second half of calendar year 2010, some companies actually start the fiscal year 2011 now in July, regardless of which scenario are you in the fact of the matter is that during this time of the year many companies are getting ready to overcome currently challenges, re-evaluate the current technologies in place and analyze migration options across the core platforms. In recently conversations that I had with ISA Server admins I notice that there are some common scenarios where ISA Server capabilities are limiting the company to achieve their business goal. The good thing is that TMG can indeed help on that.
On this post I will enumerate the top 5 scenarios where TMG can overcome ISA limitations in order to achieve the desired goal.
Scenario 1 - Consolidation and Dynamic Control
“Currently I have ISA Server 2006 that works pretty well on my company, but as our business is growing and the IT people in my team are shirking I would like centralize many things on the perimeter in order to facilitate the administration. For inbound scenario my main problem is E-Mail, it will be cool if I could have one single server to manage firewall policies as well as Exchange spam filtering capabilities. For outbound scenario I would like to have more control over the sites that my users can access in a dynamic manner, I’m can’t keep up with all suspicious site and add them to my Block List, which is a URL Set that I created. There is anything on ISA that can help me with that?”
Solution
No, ISA can’t help too much here. However, with TMG you can integrate Forefront Protection for Exchange, Exchange Edge and TMG itself in a single box. The combination of those products will allows you to implement the E-Mail Protection feature and manage all the policies in single location, which is TMG console. For your outbound challenge, TMG can indeed help here. You can use URL Filtering capability that uses MRS in order to dynamically categorize URLs that your users will access. Yes, this is the end of your endless attempt to keep up will all sites on the URL Set.
Scenario 2 – Protection against Malware
“Recently we got hit by a malware, it was pretty bad but we were able to contain the damage and cure all affected machines. After that we started a post mortem analysis to understand how this happened since all corp net workstations have anti virus, sadly we found the breach. Our guest network was not enforcing that guest computers have antivirus and I remember why we disabled that enforcement there, it was a political decision. The problem is that I have no idea if the user brought the malware or got this malware while browsing Internet through our proxy. We now need to be able to have a type of protection on the edge that can block attempts to download malicious content and help to protect unmanaged workstations that have no antivrus. Not sure if ISA can do that, please advise.”
No, ISA cannot do that. This is actually a very strong point on TMG Malware Inspection feature. With this feature enabled you can keep up the latest signature, regardless of the client workstation state (managed or unmanaged). If TMG detects an attempt to download a file that is infected, TMG will try to clean this file and if it can’t clean it will block the access to it (according to administrator's choice). The user name, file name, threat and URL will be stored on the TMG logging and you can quickly identify who attempted to download the infected file and the site that the user was trying to download it from. Yeah, I know, it’s awesome.
Scenario 3 – Keep up with the updates
“My currently ISA deployment it’s is using the 3-Leg template, I have some servers on the DMZ. Those servers are highly utilized and I’m having hard time to keep up them updated based on the monthly patch Tuesday Microsoft update cycle. The whole change request process to install new updates on the server plus the request to restart the server can take up to two weeks, in order words: my servers that live on the DMZ can be out of date for up to two weeks. In a recently internal auditing process the auditors saw that breach and we need to come up with a solution where we can mitigate that without reduce the two weeks gap that we have to apply security patches. Can ISA help us on that?”
ISA will not be able to help you to achieve this goal but TMG will. With TMG Network Inspection System you can mitigate known Microsoft vulnerabilities from being exploited via a traffic that cross TMG networks. NIS will grab the updates from Microsoft Update Service and will inspect all traffic that cross TMG, since your servers are on DMZ, NIS will evaluate traffic that are going to the DMZ (or coming from the DMZ) and verify if that traffic matches with any NIS signature, if it does and the action is set up to block, TMG will block the traffic and trigger an alert so you can easily identify a potential exploitation attempt. Now this is cool, isn’t it?
Scenario 4 – Controlling Remote Users
“We just migrate all of our domain to Windows Server 2008 and we are now implementing NAP. Since our VPN solution is based on ISA Server 2006 I would like to integrate NAP with ISA 2006, can I do that? Also, we want to allow user to connect to our VPN via SSTP. Does ISA supports SSTP?”
ISA does not integrate with NAP neither offer built in SSTP capabilities, good thing is that TMG does both. With TMG you will take advantage of Windows Server 2008 x64 bits platform which is much more robust and will be able to natively integrate with NAP via TMG Console. On top of that, TMG will also be able to help you to enable users to connect via VPN using SSTP protocol since this feature comes built in with the product. “Two birds with with single stone”, this is what I’m talking about.
Scenario 5 – We can’t stop
“Our company is growing in a fast pace, which is great, but we are becoming more and more dependent on the Internet. Recently we had an outage on our Internet connectivity with our ISP because our border router broke and we had to replace it. This replacement took two hours, it was a chaos in our company without Internet connectivity. Since this day my manager is under pressure to implement a backup plan so we have fault tolerance Internet connectivity in case the main connectivity with our ISP goes down again. I want to use ISA 2006 for that, but I’m not sure how. Any clue on how to do that?”
ISA Server 2006 doesn’t offer a built in ISP Redundancy capability that can assist you on that, but TMG does. With the new ISP Redundancy capability on TMG you can have two paths to the Internet that can be used as fail over mechanism or load balancing mechanism. This will allow you to achieve your goal and be up and running with Internet connectivity in a matter of seconds if your main ISP goes down. You’re welcome.
These are only 5 of many other scenarios that TMG can assist you to overcome the challenges that your company might be facing to keep up the business running in a secure manner. If you have ISA Server 2006, 2004 or even the almost dead ISA 2000 (extend support finishes April 2011) you should be planning your TMG migration and I will remember you again: chapter 6 of the TMG Book is your friend for that.