I’m not sure if you notice, but the Windows 8 Security Guide is already available:
It comes with the SCM 3.0 Beta (watch this interview for more information on SCM 3.0 Beta) and download it from here. Once you install this tool, browse through the path below and download the DOC file:
Note: the Windows Server2012 Security Guide is also there, under Windows Server 2012 option.
Today I have two good news about our upcoming Windows Server 2012 Security Book. Last month Deb Shinder joined us as a co-author of this book, she is already producing some great piece of content and we are just very happy to have her onboard. The other good news is that we reached 70% of the book, we are getting very close to the end (probably will be done writing by December).
Today at RSA Conference in Europe, Microsoft launched the Cloud Security Reediness Tool. Here how it works:
Go check it out now at: http://technet.microsoft.com/en-us/security/jj554736
The book that I co-wrote about Security+ is now available in Portuguese/Brazil and on the same week that the book was announced, CompTIA also announced that the Security+ Certification is now also available in Portuguese. The CompTIA Press Release from last week has a brief interview where I explain more about the book, more info here: http://www.comptia.org/news/pressreleases/13-02-20/CompTIA_Security_Certification_Exam_Now_Available_in_Portuguese_Language_Version.aspx
Consider a scenario where the UAG administrator just published OWA located on an Exchange Server 2010 and when external users click the OWA link from the portal they get the following page:
They users are able to access OWA internally without any problem and before implementing Exchange 2010, the users were able to access OWA using Exchange 2007.
Reviewing the Web Monitor log it is possible to notice the following errors:
As you can see the error message that is highlighted above says that the application Exchange 2010 of type ExchangePub2007 failed. This means that when the administrator created the Exchange publishing rule within the trunk he selected the wrong Exchange version (or in this case he just used the same application in the portal after replacing Exchange 2007 to Exchange 2010). This option is available on step 2 of the add application wizard as shown below:
When publishing Exchange through UAG 2010 make sure to choose the correct version on step 2 of the Add Application Wizard and if you currently have a trunk that is publishing Exchange 2007, do not change the rule to point to a new Exchange 2010 Server, you will need to create a new application publishing within the trunk to publish Exchange 2010.
I was reading the Windows IT Pro Magazine of this month (September 2009) and there I found a nice article written by an Escalation Engineer here from Microsoft Texas (Michael Morales) where he describes how to use ProcDump to catch high CPU utilization. This is an amazing tool that can also help ISA Administrators, mainly for scenarios where we just can’t get the right data (most case dumps) because the issue is random and when it happens there is nobody available to execute a command (for example: launch DebugDiag and choose the option for manual dump the process).
For an ISA Server high CPU utilization scenario a simple example will be dump out the Firewall Service process two times when the CPU for wspsrv.exe is at or exceeds 90 percent for 5 seconds and store the dumps in the c:\dumps folder:
c:\procdump.exe -c 90 -s 5 -n 2 wspsrv.exe c:\dumps
Isn’t that cool?
Make sure to read the article from Michael Morales to fully understand how this tool works:
As new folks are starting to install Forefront TMG 2010 they are finding out that right after install it they already have an alert on Forefront TMG console similar to the one below:
This behavior is documented in the Forefront TMG 2010 Release Notes and says:
Windows Filtering Platform error message following a computer or Forefront TMG services restart
After you restart the Forefront TMG computer or services, the following error message might be displayed:“Forefront TMG detected Windows Filtering Platform filters that may cause policy conflicts on the server. The following providers may define filters that conflict with Forefront TMG firewall policy: Microsoft Corporation.”If this message is displayed, disable the alert from appearing again, since it does not indicate a real conflict.
As the release notes says this is an expected error message, it happens because Forefront TMG firewall engine detects filters on Windows Filtering Platform and it can be safely ignored. If you want to confirm that Forefront TMG is handling the core WFP categories you can use the netsh command below:
Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Users\administrator.CONTOSO>netsh advfirewall monitor show firewall
BootTimeRuleCategory Microsoft Forefront Threat Management Gateway
FirewallRuleCategory Microsoft Forefront Threat Management Gateway
StealthRuleCategory Microsoft Forefront Threat Management Gateway
ConSecRuleRuleCategory Windows Firewall
For more information on TMG integration with WFP read Chapter 1 (page 7) of the Microsoft Press Forefront TMG Administrator’s Companion book.
The video that me and Tom Shinder recorded is now available at Edge web site. On this 23 minutes video we talk about the Web Protection on Forefront TMG and the features that are part of this solution.
A special thanks to my friend Mohit Kumar (Sr. SEE on ISA Team) for interviewing us and for David Tesar to publish this at Edge Web site.
We just released this hotfix for Forefront TMG 2010, which cover issues involving integrated NLB feature on TMG, mainly when used in VPN Scenarios (Site to Site for example). More information about this fix go to the link below:
Many companies are starting to budget for the second half of calendar year 2010, some companies actually start the fiscal year 2011 now in July, regardless of which scenario are you in the fact of the matter is that during this time of the year many companies are getting ready to overcome currently challenges, re-evaluate the current technologies in place and analyze migration options across the core platforms. In recently conversations that I had with ISA Server admins I notice that there are some common scenarios where ISA Server capabilities are limiting the company to achieve their business goal. The good thing is that TMG can indeed help on that.
On this post I will enumerate the top 5 scenarios where TMG can overcome ISA limitations in order to achieve the desired goal.
Scenario 1 - Consolidation and Dynamic Control
“Currently I have ISA Server 2006 that works pretty well on my company, but as our business is growing and the IT people in my team are shirking I would like centralize many things on the perimeter in order to facilitate the administration. For inbound scenario my main problem is E-Mail, it will be cool if I could have one single server to manage firewall policies as well as Exchange spam filtering capabilities. For outbound scenario I would like to have more control over the sites that my users can access in a dynamic manner, I’m can’t keep up with all suspicious site and add them to my Block List, which is a URL Set that I created. There is anything on ISA that can help me with that?”
No, ISA can’t help too much here. However, with TMG you can integrate Forefront Protection for Exchange, Exchange Edge and TMG itself in a single box. The combination of those products will allows you to implement the E-Mail Protection feature and manage all the policies in single location, which is TMG console. For your outbound challenge, TMG can indeed help here. You can use URL Filtering capability that uses MRS in order to dynamically categorize URLs that your users will access. Yes, this is the end of your endless attempt to keep up will all sites on the URL Set.
Scenario 2 – Protection against Malware
“Recently we got hit by a malware, it was pretty bad but we were able to contain the damage and cure all affected machines. After that we started a post mortem analysis to understand how this happened since all corp net workstations have anti virus, sadly we found the breach. Our guest network was not enforcing that guest computers have antivirus and I remember why we disabled that enforcement there, it was a political decision. The problem is that I have no idea if the user brought the malware or got this malware while browsing Internet through our proxy. We now need to be able to have a type of protection on the edge that can block attempts to download malicious content and help to protect unmanaged workstations that have no antivrus. Not sure if ISA can do that, please advise.”
No, ISA cannot do that. This is actually a very strong point on TMG Malware Inspection feature. With this feature enabled you can keep up the latest signature, regardless of the client workstation state (managed or unmanaged). If TMG detects an attempt to download a file that is infected, TMG will try to clean this file and if it can’t clean it will block the access to it (according to administrator's choice). The user name, file name, threat and URL will be stored on the TMG logging and you can quickly identify who attempted to download the infected file and the site that the user was trying to download it from. Yeah, I know, it’s awesome.
Scenario 3 – Keep up with the updates
“My currently ISA deployment it’s is using the 3-Leg template, I have some servers on the DMZ. Those servers are highly utilized and I’m having hard time to keep up them updated based on the monthly patch Tuesday Microsoft update cycle. The whole change request process to install new updates on the server plus the request to restart the server can take up to two weeks, in order words: my servers that live on the DMZ can be out of date for up to two weeks. In a recently internal auditing process the auditors saw that breach and we need to come up with a solution where we can mitigate that without reduce the two weeks gap that we have to apply security patches. Can ISA help us on that?”
ISA will not be able to help you to achieve this goal but TMG will. With TMG Network Inspection System you can mitigate known Microsoft vulnerabilities from being exploited via a traffic that cross TMG networks. NIS will grab the updates from Microsoft Update Service and will inspect all traffic that cross TMG, since your servers are on DMZ, NIS will evaluate traffic that are going to the DMZ (or coming from the DMZ) and verify if that traffic matches with any NIS signature, if it does and the action is set up to block, TMG will block the traffic and trigger an alert so you can easily identify a potential exploitation attempt. Now this is cool, isn’t it?
Scenario 4 – Controlling Remote Users
“We just migrate all of our domain to Windows Server 2008 and we are now implementing NAP. Since our VPN solution is based on ISA Server 2006 I would like to integrate NAP with ISA 2006, can I do that? Also, we want to allow user to connect to our VPN via SSTP. Does ISA supports SSTP?”
ISA does not integrate with NAP neither offer built in SSTP capabilities, good thing is that TMG does both. With TMG you will take advantage of Windows Server 2008 x64 bits platform which is much more robust and will be able to natively integrate with NAP via TMG Console. On top of that, TMG will also be able to help you to enable users to connect via VPN using SSTP protocol since this feature comes built in with the product. “Two birds with with single stone”, this is what I’m talking about.
Scenario 5 – We can’t stop
“Our company is growing in a fast pace, which is great, but we are becoming more and more dependent on the Internet. Recently we had an outage on our Internet connectivity with our ISP because our border router broke and we had to replace it. This replacement took two hours, it was a chaos in our company without Internet connectivity. Since this day my manager is under pressure to implement a backup plan so we have fault tolerance Internet connectivity in case the main connectivity with our ISP goes down again. I want to use ISA 2006 for that, but I’m not sure how. Any clue on how to do that?”
ISA Server 2006 doesn’t offer a built in ISP Redundancy capability that can assist you on that, but TMG does. With the new ISP Redundancy capability on TMG you can have two paths to the Internet that can be used as fail over mechanism or load balancing mechanism. This will allow you to achieve your goal and be up and running with Internet connectivity in a matter of seconds if your main ISP goes down. You’re welcome.
These are only 5 of many other scenarios that TMG can assist you to overcome the challenges that your company might be facing to keep up the business running in a secure manner. If you have ISA Server 2006, 2004 or even the almost dead ISA 2000 (extend support finishes April 2011) you should be planning your TMG migration and I will remember you again: chapter 6 of the TMG Book is your friend for that.
Recently Tom Shinder published two very useful and well explained (as usual) articles about TMG ISP Redundancy. This is a new TMG feature that for years ISA administrators were looking for and I’m sure you will be very happy with the end results for this feature on TMG. But, before implement it is good to read through the article to understand how it works. Visit the links below for more info on Tom’s article:
I recently worked in a very interesting case where customer’s Exchange Server got in the SPAM Block list although the environment was clear of malware and no SPAM was originated from that server at all. We ended up identifying why the server got blocked and it was because an external servers was using reverse DNS lookup to verify if the MX record for that email server matches with the source IP address from where the SMTP traffic was coming from. To make it easier to understand, let’s take a look on following diagram for contoso.com network:
Notice that the primary IP bound to ISA’s external interface is using IP 192.168.1.113. The SMTP Publishing rule correctly maps the internal Exchange Server IP but the outbound traffic always will leave with the primary IP of the ISA Server. This means that when the external Exchange Server performs the reverse lookup for the MX record (for example: mail.contoso.com) it will resolve for 192.168.1.60 which doesn’t match with the source IP received in the IP header of the SMTP packet.
The fast resolution here is to change the primary IP to be 192.168.1.60, but sometimes this cannot be done so fast due other policies for example. But….that’s the way it is on ISA Server, not much you can do other than plan to use the primary IP for scenarios like this.
The good thing here is: TMG resolves this problem! How? With a feature called Enhanced NAT (ENAT). Now you can create a network rule to specify which IP address you want to use for outbound traffic as shown below:
Isn’t that nice? It’s amazing for sure!!
Well, we are already in September and TMG is coming very soon…but while is not RTM yet, you still have a chance to download Beta 3 and play with it.
As we now have Exchange 2010 RC available for download, many of you that are testing Forefront TMG 2010 RC are asking if you can test the Email Protection feature using Exchange 2010. If you read the paper Understanding E-Mail Protection on Forefront TMG published at Tales from the Edge, you will see that one of the questions in the Q&A is:
Question 12) Which versions of Exchange do you support?
Answer: We support Exchange Edge 2007 SP2 and Exchange Edge 2010.
With that you know that it is supported, but the opening question is: how to install Exchange 2010 Edge role and Forefront Protection 2010 on top of an existing Forefront TMG 2010 RC installation? This is exactly the goal of this post; guide you through the steps to perform this installation. This post is assuming that TMG 2010 RC is running on Windows Server 2008 R2.
2. Preparing the Environment for Exchange 2010
Before install Exchange 2010 RC you should install a series of prerequisites and the best way to do this is by following the guidelines from Exchange 2010 Prerequisites document, under the section Install the Windows Server 2008 R2 operating system prerequisites. After complete this process, than you can run the Exchange 2010 setup and choose the following options:
1. Select Install Microsoft Exchange as shown below:
Figure 1 – Selecting Exchange setup option.
Figure 1 – Selecting Exchange setup option.
2. Click Next in the Introduction page. Read the license agreement, select I accept the terms in the license agreement and click Next to continue.
3. Select Yes in the Error Reporting page and click Next.
4. Select Custom Exchange Server Installation as shown below and click Next to proceed:
Figure 2 – Selecting Custom installation.
5. Select Edge Transport Role in the Server Role selection as shown below and click Next to continue:
Figure 3 – Selecting Edge Transport Role.
6. Chose the appropriated option for the CEIP and click Next to continue.
7. Wait until the readiness check finishes and when your window appears as shown below click Install to proceed:
Figure 4 – Click Install to proceed.
8. When the setup finishes as shown in the figure below, uncheck the option Finalize Installation using the Microsoft Exchange Console and click Finish button to conclude the process.
Figure 5 – Setup finished.
9. On Exchange Setup window, click step 5 – Get Critical update for Microsoft Exchange.
10. Install any critical update that it might have and close the Exchange Setup window.
At this point you already have Exchange 2010 Edge Role installed on your system; next step is to install Forefront Security 2010 for Exchange on TMG.
3. Running Exchange Installation via TMG 2010 Setup
Follow the steps below to install Forefront Security 2010 for Exchange from the TMG setup:
1. Execute the autorun.hta file and choose the option to Install Microsoft Forefront Protection 2010 for Exchange Server:
Figure 6 – Choose the option to install Forefront Security 2010 for Exchange.
2. Select to the terms of the license agreement and privacy statement and click Next.
3. You should receive a notification saying that Exchange Transport service will be restarted. Click Next to proceed.
4. Confirm the installation folders (or change according to your preference) and click Next.
5. Click Next on the Proxy configuration.
6. Leave the Enable antispam now option selected as shown in Figure below and click Next to proceed:
Figure 7 – Enabling Antispam.
7. Leave the Enable antispam now option selected as shown in Figure below and click Next to proceed:
8. Chose the appropriated option for the CEIP and click Next to continue.
9. Review all your selections in the Confirm Settings page as shown below and click Next to continue:
Figure 8 – Reviewing installation settings.
10. While the installing is happening you will also see the window below saying that the setup is configuring the product and services:
Figure 9 – Configuring product and services setup window.
11. After that you should see the last setup window saying that the installation finished successfully as shown below:
Figure 10 – Reviewing installation results.
12. Click Finish button to finish the setup.
13. Click Exit to close the TMG Setup window.
Now you have both consoles available: Exchange and Forefront Protection 2010 for Exchange as shown below:
Figure 11 – FSE and Exchange console available after finishing this procedure.
Note: Something to keep in mind: changes that you perform on TMG 2010 regarding Email Protection will be applied to Exchange Edge and FSE according to the option that you choose. Read the paper Understanding E-Mail Protection on Forefront TMG published at Tales from the Edge for more information on what feature each product owns.
In this post you learned how to install Exchage 2010 Edge Role and Forefront Protection 2010 Beta for Exchange on top of an existing Forefront TMG 2010 RC installation. Now that the setup is done, use the Configuring protection from e-mail-based threats article to configure this feature.
After working on this article for a couple of weeks on my spare time this Troubleshooting Survival Guide is now ready for you, check it out at:
As you know, this is a Wiki article and you can contribute with it, but first take a look and see if you like it
Ever wonder what happened to the security guides from Microsoft Solution Accelerators? Your go-to security guidance from Solution Accelerators hasn’t disappeared, it’s just been repackaged. The previously stand-alone Microsoft product-specific security guides are now included within the Microsoft Security Compliance Manager (SCM) tool.
To simplify: Stand-alone security guides à Security Compliance Management Toolkit à Security Compliance Manager tool
So, how do you get your hands on trusted guidance for Windows client and server operating systems and Microsoft applications from the Microsoft security experts? First download and install the SCM tool. Next, import your product baselines of choice (options include Windows 7, Windows Server 2008 R2, Internet Explorer 8, and more). Finally, select the Documents tab within a baseline to access the security guide for that Microsoft product. Simple, right?
SCM is just one of the tools provided by the Microsoft Solution Accelerators team. The Microsoft Assessment and Planning Toolkit, Microsoft Deployment Toolkit, and Security Compliance Manager provide tested guidance and automated tools to help you plan, securely deploy, and manage new Microsoft technologies—easier, faster, and at less cost. All are freely available, and fully-supported by Microsoft. Learn more.
Before joining the Windows iX IT PRO Security team I spent my last 11 years working in the enterprise support field, where 5 were at Microsoft CSS (former PSS). During the Conficker outbreak I was in Oklahoma for New Years Eve 2008/2009 (which BTW is pretty cool) and while I was there I wrote this post about blocking Conficker proliferation via ISA Server. Six months later I kept hearing comments from the IR (Incident Response) folks about getting new cases related to Conficker. I remember talking to one of those guys and hearing from him that some companies were without patches for years. We are not talking about small offices; I’m talking about enterprise level type of company with thousands of workstations and hundreds of servers - unpatched.
While it is hard to believe that this type of practice still happening, today reading this article I got the confirmation that Conficker didn’t teach the full lesson to everybody. Unfortunately the reality is that there are still many servers and workstations (regardless of the OS) unpatched out there. In other hand, it is also very good to see that people are warning about that in many ways, such as with an article like this: “Patch Management Crucial to Defend Against Cyber-Attacks: Report” that explains how important it is to patch and beyond that, how important it is to make sure all platforms are patched. The article has a great statement, that says:
“While Windows vulnerabilities receive wide attention, Norman security experts also warned that IT administrators in enterprises, government and small to midsize businesses (SMBs) should focus on patch management involving all major operating systems, including Microsoft Windows, Linux, Mac OS, Sun Solaris and HP.”
“While Windows vulnerabilities receive wide attention, Norman security experts also warned that IT administrators in enterprises, government and small to midsize businesses (SMBs) should focus on patch management involving all major operating systems, including Microsoft Windows, Linux, Mac OS, Sun Solaris and HP.”
If you don’t know where to start on patch management subject for your Microsoft platform or if you want to review if you patch management strategy is correct, go ahead and download the Microsoft Security Update Guide, Second Edition – this is a great source of information about this subject.
I just want to drop a quick note to all Security MVPs that were able to join this morning in the webcast below:
It was great meeting you and sharing some details of our community plan. I’m really excited with what’s coming and as I said you have a big role and the capability of making a huge impact. Looking forward to work with you even more!
As soon as the recording session is available I will drop a note here as well.
Hi Folks, I just want to drop a quick note here about KB http://support.microsoft.com/kb/2433623/ that brings the list of the updates that are part of the new Software Update 2 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 1.
Yes, another case where TMG stops responding…bad, bad TMG right? NOT!!! Recently I worked in some scenarios where TMG was stopping every day, during the same time and required a manual restart. TMG Admin claimed that nothing really changed on TMG or on the client workstations, besides the environment was running rock solid for a long time and the issue started happened couple of weeks ago.
2. Digging In
After many sessions of data gathering (believe me, it can take more than one round to find out) using the usual approach to collect performance related data I found out the following trend when the issue was happening:
This is the Forefront TMG Firewall Packet Engine\Backloggged Packets counter, which should be 10 and it was 2,485.000 (WOW…just WOW). Notice this beautiful line going from 0 to 2K and worst, staying there forever. Now we know why TMG stops responding, but why backlogging is growing? Well, there are two core reasons: authentication and/or name resolution. On the user mode dump of wspsrv.exe we have hundreds of threads like this:
Child-SP RetAddr Call Site
00000000`11fbccc8 000007fe`fd82aa76 ntdll!ZwAlpcSendWaitReceivePort+0xa
00000000`11fbccd0 000007fe`fd8ccb64 rpcrt4!NdrDllCanUnloadNow+0x31c6
00000000`11fbcd90 000007fe`fd8ccd55 rpcrt4!Ndr64AsyncClientCall+0xe04
00000000`11fbd050 000007fe`fc9e1f95 rpcrt4!NdrClientCall3+0xf5
00000000`11fbd3e0 000007fe`fc9e1e74 dnsapi!DnsApiAlloc+0xdd1
00000000`11fbd440 000007fe`fc9e60a6 dnsapi!DnsApiAlloc+0xcb0
00000000`11fbd500 000007fe`fca0d012 dnsapi!DnsValidateName_W+0x186
00000000`11fbd580 00000000`72cab68f dnsapi!DnsQuery_A+0x36
00000000`11fbd5d0 00000000`72caaced msphlpr!COC_NameResolution_TargetImpl::FoundInNegativeCache+0x2b93
00000000`11fbd6a0 00000000`72ca6efe msphlpr!COC_NameResolution_TargetImpl::FoundInNegativeCache+0x21f1
00000000`11fbd960 00000001`3fa467ff msphlpr!ProxyGetHostByAddr+0x4a2
00000000`11fbdce0 00000001`3fa48420 wspsrv!FwGapaGetConfig+0x4b4a3
00000000`11fbddb0 00000001`3f8ded8d wspsrv!FwGapaGetConfig+0x4d0c4
00000000`11fbe660 00000001`3f92e07c wspsrv+0x5ed8d
00000000`11fbe730 00000001`3f92d79e wspsrv!IsChainingRequired+0x163cc
00000000`11fbef90 00000001`3f8ea240 wspsrv!IsChainingRequired+0x15aee
00000000`11fbf050 00000001`3f8f1c0a wspsrv+0x6a240
00000000`11fbf1f0 00000001`3f8f11d2 wspsrv+0x71c0a
00000000`11fbf270 00000001`3f9838bf wspsrv+0x711d2
00000000`11fbf320 00000001`3f97d871 wspsrv!DeleteFwEngFilter+0x249b
00000000`11fbf360 00000001`3fa1bedc wspsrv!IsChainingRequired+0x65bc1
00000000`11fbf550 00000001`3f971a53 wspsrv!FwGapaGetConfig+0x20b80
00000000`11fbf6a0 00000001`3f94185c wspsrv!IsChainingRequired+0x59da3
00000000`11fbf780 00000001`3f9415ce wspsrv!IsChainingRequired+0x29bac
00000000`11fbf7f0 00000000`771cf56d wspsrv!IsChainingRequired+0x2991e
00000000`11fbf880 00000000`77403021 kernel32!BaseThreadInitThunk+0xd
00000000`11fbf8b0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
…and lots more like this:
00000000`021aece8 000007fe`fd3e10ac ntdll!ZwWaitForSingleObject+0xa
00000000`021aecf0 00000001`3f90609b KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`021aed90 00000001`3f8ec283 wspsrv+0x8609b
00000000`021aee60 00000001`3f8f1c37 wspsrv+0x6c283
00000000`021af250 00000001`3f8f11d2 wspsrv+0x71c37
00000000`021af2d0 00000001`3f9838bf wspsrv+0x711d2
00000000`021af380 00000001`3f97d9fe wspsrv!DeleteFwEngFilter+0x249b
00000000`021af3c0 00000001`3fa1bedc wspsrv!IsChainingRequired+0x65d4e
00000000`021af5b0 00000001`3f971a53 wspsrv!FwGapaGetConfig+0x20b80
00000000`021af700 00000001`3f94185c wspsrv!IsChainingRequired+0x59da3
00000000`021af7e0 00000001`3f9415ce wspsrv!IsChainingRequired+0x29bac
00000000`021af850 00000000`771cf56d wspsrv!IsChainingRequired+0x2991e
00000000`021af8e0 00000000`77403021 kernel32!BaseThreadInitThunk+0xd
00000000`021af910 00000000`00000000 ntdll!RtlUserThreadStart+0x21
I cannot use the private symbols here (for obvious reasons – they are private ), but this function is dealing with re-injection and we had 50 threads performing this operation. This is a magic number, because 50 is the default value of re-injection threads on TMG as I explain here.
3. Moving forward
We now know why TMG box hangs, but why we have this gigantic amount of authentication if the environment didn’t suffer any change, the applications are the same, the users are the same…how’s that possible? Maybe a malware sending burst traffic from inside to outside? We didn’t know but we continued the investigation and concluded that the environment was clean from malware.
We used netmon to understand from where the traffic was coming from and it was identified some IPs on the internal network that were sending this gigantic amount of traffic. We tracked that IPs and found the owners of those computers; they were contractors that were in the company performing a project. Guess what? They were using a P2P application to download “some stuff”. I started reviewing more info about this application and found the following statement on their website:
“If you use a software firewall (e.g. ZoneAlarm) you will need to make sure Ares P2P gets full and unlimited access to the Internet.” Source: http://www.aresp2p.net
What’s your reading on this? I will let you think about that.
This was not a TMG problem, TMG was hanging because it was waiting from DC/DNS to reply to the plethora of requests that it was sending through the network and as it couldn’t authenticate the users and couldn’t allow them just to go through. We really need to step back here and think about security in a broader manner, how do you validate the guest computers on your corporate network? Because here, the environment had a good security policy for domain joined computers by using software restriction policy and disallowing non-authorized applications to run on the corporate environment. But, it didn’t have a validation process for guest computers. I will suggest start by reading this “Protect Corporate Assets from Unmanaged Computers” and go from there. At the end of the day you don’t want to create a rule on your firewall allowing everything for all users just because there are some applications that need full Internet access, unless you are willing to take the risk for such action, are you?
(Note: read the follow up of this post here).
Yesterday I published this post about an issue that caused TMG stop to respond and I want to clarify one key point here: TMG didn’t stop because was not able to handle the load, let’s be clear on that. Maybe it was not clear for some readers that don’t know about how TMG works, but the issue here was that the DC was not able to handle the gigantic amount of authentication request on the time speed that TMG was sending the requests and waiting for an answer. As a result of that TMG’s backlog started to grow and caused this behavior. Its plain simple: DC was not sized to handle that amount of authentication request. Again, not a TMG issue.
Couple of things can be done to avoid that those incidents don’t fully affect your environment. Here are some key tips (nothing new, but maybe you missed):
· Don’t create rules allowing ALL OUTBOND TRAFFIC as Protocol. This may cause issues as I explained in this post.
· Make sure to use Internet Explorer 7 or higher to take advantage of Kerberos, which will distributed the authentication load among the DCs. Back in 2008 I wrote this article that explains in details all the advantages of using Kerberos for Proxy authentication.
By using those practices you offload the authentication request to go from TMG to the DC and leave this task for the workstation (again read this article for more info), which dramatically impact the backlog (by lowering the utilization). Last but not least I want to say that it’s all about sizing: if the environment was sized to receive 20 x 100, it will have a negative impact if you see 2000 x 100. There is no magic here, in this case TMG was correctly sized, but as a secure firewall it couldn’t allow traffic to pass through without waiting for the DC to reply back saying that that request comes from a valid user, therefore it will fail safe and block the traffic from traversing the networks.
BTW, for those of you that still believe that Hardware Firewall is better, I will let you with the wise words of my friend Tom Shinder about this old discussion: Tom Shinder on “Hardware” Firewalls.
Almost one month without update, that’s very rare for sure, time is working against me but pretty soon I should be able to breathe again. Last couple of months I was studying to get my CEH certification and couple of weeks ago and I got it. Here are some resources that I recommend you to read if you are studying for this certification:
Some Articles to Complement
Now I’m working to finish my new book (in Portuguese only) about Security+ Certification (Portuguese’s readers go to www.securityplusbr.org for more info). This project should be done by end of this month and this book should be released in Brazil April next year.
On the community space my last contributions were:
My MVP friends are very active these days and here are some great articles that they recently published:
Next week I should have new posts here with some interesting issues that I’m dealing with.
Today I was assisting a friend of mine here from TMG team that was facing this issue, same issue that was also mentioned on this thread. The problem was happening when using Cryptography Next Generation (CNG) or also called V3, TMG was not recognizing the private key and was showing up this error message. This is a known issue because TMG (and ISA) don’t support CNG (V3 Certificates). This is well documented under the unsupported documentation here:
Issue: Forefront TMG does not support the use of certificates created using CNG (Certificate New Generation) based templates for Web listeners or as client certificate authentication in Web publishing or Web chaining rules.
Cause: CNG certificates are not usable by Forefront TMG.
Workaround: Create certificates using Windows 2000 or Windows 2003 templates.
Again, make sure to read this unsupported document before deploy TMG, there you will find the official statement from TMG Product Team about what it is supposed to work and what it is not.
Note: Important to emphasize that CNG V3 is not X.509 V3. CNG V3 refers to the new V3 Certificate Template on 2008 while X.509 V3 is the current certificate standard in which TMG is fully compatible.
I will be delivering two sessions at TechEd Brazil this year, one about TMG and another one about IAG 2007. Steve Riley also is going to deliver four sessions and Steve Ballmer will present the keynote. It is going to be an amazing event and if you did not register yet it is better do it quickly at http://www.teched.com.br .
I know some friends from Brazil, Argentina and Chile are going, so I hope to see you guys there. J
Now that TMG Beta 3 is released you can enjoy the best of both words for VPN access. In the past I was questioned about SSTP on ISA Server 2006 since Windows Server 2008 was capable to do it. The sad answer was that ISA Server 2006 didn’t have this feature built in. But now you can use TMG and select SSTP the same way as another protocol as shown in Figure 1:
Figure 1 – SSTP available in TMG Console.
When configuring SSTP on TMG you will need to carefully plan:
· Web Listener that will be used by SSTP.
· Certificate that is going to be bound to the Web Listener.
Besides that you will need Windows Vista with SP1 on the client workstation to test this new feature.
Troubleshooting Client Access
Since I’m working remotely some these days I was able to reproduce some of the nice errors that I didn’t have when I was in my home lab. Today for example I got the following error when I was trying to connect from my laptop:
Figure 2 – First error due the cert name.
That was pretty self explanatory, but just to confirm the name that I used to issue the certificate I got a netmon trace and got the subject name:
SSL: Server Hello. Certificate. Server Hello Done.
Seq=1878717387 - 1878718743, Ack=2650000305, Win=256 (scale factor 0x8) = 65536
- Ssl: Server Hello. Certificate. Server Hello Done.
+ Version: TLS 1.0
Length: 1351 (0x547)
- SSLHandshake: SSL HandShake TLS 1.0 Server Hello Done(0x0E)
Length: 70 (0x46)
+ ServerHello: 0x1
Length: 1269 (0x4F5)
- Cert: 0x1
CertOffset: 1266 (0x4F2)
CertificateLength: 1263 (0x4EF)
- X509Cert: Issuer: contoso-DC01-CA,contoso,com, Subject: vpn.contoso.com,IT,Contoso,Dallas,Texas
- TbsCertificate: Issuer: contoso-DC01-CA,contoso,com, Subject: vpn.contoso.com,IT,Contoso,Dallas,Texas
+ Version: v3 (2)
+ SerialNumber: 0x6168a464000000000002
+ Signature: Sha1WithRSAEncryption (1.2.840.1135184.108.40.206)
+ Issuer: contoso-DC01-CA,contoso,com
+ Validity: From: 06/15/09 21:03:46 UTC To: 06/15/10 21:13:46 UTC
+ Subject: vpn.contoso.com,IT,Contoso,Dallas,Texas
+ SubjectPublicKeyInfo: RsaEncryption (1.2.840.1135220.127.116.11)
+ SignatureAlgorithm: Sha1WithRSAEncryption (1.2.840.113518.104.22.168)
HandShakeType: Server Hello Done(0x0E)
Length: 0 (0x0)
To quick fix this I edited my host file and created a manual entry there. But then right after that I got:
Figure 3 – Now is the CRL.
Looking to the properties of the certificate it was possible to see that the CRL was poiting to my internal CA:
Figure 4 – The CRL for my internal CA.
To resolve this I created a web publishing rule to publish my CRL and after that all worked fine.
While testing those settings I got some great links from the RRAS team (which is the component that TMG uses for VPN capability). Check it out the links below:
You might be wondering: how did you get access to those things if you were unable to establish the VPN connection? The answer is: through my backup PPTP connection :)
Error 64 can happen due many situations and I documented one of those situations last year and as you could see sometimes it is not easy to find out why this error happens. The issue that I’m about to describe here was identified while I was troubleshooting a third party application that uses TCP Port 80 to transmit files, but not using HTTP. What?? Yeah, I know. Although IANA has established port 80 for HTTP, anyone can create an application that uses port 80 to send whatever they want. This is fine, as long as you don’t try to use this application behind a Firewall that does application layer inspection and look to that traffic and say: what is that? This is not HTTP Protocol and it is using TCP Port 80…I shall block this traffic!
The firewall administrator was smart to understand that and what he did was, he created a custom protocol using port 80 and didn’t bind Web Proxy filtering to it. Fair enough, but doesn’t fully resolved this issue.
2. The Error
When the client (which had the 3rd party application installed on his computer) started to transmit the file to the destination it received an error and didn’t proceed. Using Logging feature the Firewall Administrator saw the error below:
Figure 1 – Error 64
On the netmon trace we could see that the TCP Handshake was established fine, but after the first HTTP Payload has being sent ISA Server 2006 didn’t like what he saw and the connection was reset.
Figure 2 – Connection reset right after first attempt to use TCP Port 80 (with a non compliance HTTP Protocol).
To resolve this problem what you need to do is not only create a custom protocol and an access rule to use this protocol, but also a deny rule right after this access rule to block the regular HTTP Protocol that has the Web Proxy Filter bind to it. The access rules will look like this:
Figure 3 – Access rule with a Deny to HTTP (with filter) Protocol.
Why do I have to do this? Read this post here and you will know the reason:
Why do I need a deny rule to make an allow rule for a custom protocol work correctly?