1. Introduction

This post is about a scenario where Firewall Administrator notices that the application log in the ISA Server computer is full of events 14197 saying:

He notices that after some time with this event one of the following symptoms happen:

·         The Firewall Service stops responding and when he tries to restarts it hangs with starting status for awhile before the status change to started.

·         Sometimes the Firewall Service quits without any message.

·         Sometimes it is not possible to restart firewall service because it hangs in stopping status.

I’m very familiar with those events and most of the time (9 out 10) this is caused by a third party software scanning or locking the cache file, the article http://support.microsoft.com/kb/887311 explains that in details. For this particular scenario the most common third party application that can cause this is the antivirus. When you have file scan antivirus installed on ISA Server (or TMG) you need to use the article Considerations when using antivirus software on ISA Server to create an exclusion list of the folders, files and process that shouldn’t be scanned by the antivirus.

So, if I follow the recommendations from that article I should be good, right? You should, as long as the exclusions are REALLY…. REALLY in place. I’m emphasizing the “really” because of some recent unfortunate experiences that I had; let me tell you a story to chill the bones about a thing that I saw…oops, that a song.

2. The Tales

Recently I worked in at least three different scenarios very similar where the firewall administrator created the exclusion list by following that article, however the exclusions were added though AV policy deployment located in the central antivirus server management. The reason why this policy was deployed via central location was because the local client AV installed on ISA didn’t allow local changes; everything was done by this AV Server. However even after making those exclusions the event 14197 was still happening. Of course I heard the most common argument in such scenarios: I told you that this was not a problem on our AV, this is an ISA issue for sure.

Ok, at that point I couldn’t really argue and had to think that every piece of software is innocent until we prove otherwise. So I followed this line of think I went to a path of collecting evidence to see who was doing that. Hunting time !!

3. Preparing the Environment

One way to collect evidences that there is a process other than ISA Server process is to audit the folder where the cache is located, by default %systemdrive%\urlcache. To do that you need to follow the steps below:

1. Enable local audit for object access (Start / Run / secpol.msc):

1. Introduction

Recently the number of times that I received this question increased (not sure why), but the fact of the matter is that you can do something to resolve this problem (or at least identify the source of the problem), if you have the right tools. When your ISP informs you that you are in an email blacklist you can review the results by going to a site that provides real time blacklist results, such as http://www.mxtoolbox.com/blacklists.aspx.

2. Step 1 – Identifying

Next thing to do is identify the source of the problem, why your company got blacklisted? If you have ISA Server in the border of your network, what you can do is just watch your live logging creating a filter for SMTP Protocol coming from the internal network, as shown below:

If you have only one SMTP Server internally, then the only IP that you should see sending SMTP traffic is your SMTP server, if you start to see workstation’s IP address in the list then you need to investigate that further. Make sure to also get a netmon trace on ISA (internal and external adapter) at the same time or simple get an ISA Data Packager in repro mode using Web Proxy and Web Publishing template.

2. Step 2 – Containing

To contain the amount of SMTP traffic leaving your network you need to make sure that the only host that is capable to send SMTP traffic out (to the Internet) is your SMTP Server.

This is the reason why I get disappointed when I open a Firewall Policy rule and see a rule that allows All Outbound. If your environment doesn’t have such rule, the SMTP traffic that are not coming from the SMTP Server will not be able to leave your internal network in the first place.

Notice that I’m using step two as contention because on step one you are logging the real traffic, which means that you will have real data to analyze it later.

3. Step 3 – Remediating

After implementing this contention, you can now start working on the workstation and verify why it is sending traffic out to the Internet. At this point, you can:

· Get a netmon sample of the traffic, so you can see which process is sending out SMTP traffic.

· Unplug this workstation from the network

· Use an Antivirus (like Forefront Client Security or Microsoft Security Essentials) to scan the local workstation and see if it is infected.

After cleaning the system (last time that I worked in a case similar to this one, the piece of malware that was found in the workstation was Backdoor:Win32/Oderoor.gen!A.), one other thing that you can also do is to follow the procedures form the article Capturing a Trace at Boot Up and get a sample capture while the machine is starting up. With that you can see the traffic profile and make sure that there is no malicious attempt to go out right in the beginning of the system boot.

4. Conclusion

This is only a simple example of an incident response in case your SMTP is being blacklisted due a piece of malware that is running inside your network. A full description of Microsoft Incident Response recommendation read the article below:

Responding to IT Security Incidents

http://technet.microsoft.com/en-us/library/cc700825.aspx

1. Introduction

Consider a scenario where the UAG administrator just published OWA located on an Exchange Server 2010 and when external users click the OWA link from the portal they get the following page:

They users are able to access OWA internally without any problem and before implementing Exchange 2010, the users were able to access OWA using Exchange 2007.

2. Troubleshooting

Reviewing the Web Monitor log it is possible to notice the following errors:

As you can see the error message that is highlighted above says that the application Exchange 2010 of type ExchangePub2007 failed. This means that when the administrator created the Exchange publishing rule within the trunk he selected the wrong Exchange version (or in this case he just used the same application in the portal after replacing Exchange 2007 to Exchange 2010). This option is available on step 2 of the add application wizard as shown below:

3. Conclusion

When publishing Exchange through UAG 2010 make sure to choose the correct version on step 2 of the Add Application Wizard and if you currently have a trunk that is publishing Exchange 2007, do not change the rule to point to a new Exchange 2010 Server, you will need to create a new application publishing within the trunk to publish Exchange 2010.

It is very common for most customers to really wait until a product gets more mature, wait to see the trends and evaluate if they should migrate or not. Questions like: who is adopting this platform? what advantages I will have using this new product? why should I migrate my stable ISA environment to TMG?  Those are very common and understandable questions and the fact of the matter is that there is nothing better than real world case scenarios to really answer those questions. If this is your case, then take a look on the Forefront TMG 2010 case studies page at http://www.microsoft.com/forefront/threat-management-gateway/en/us/case-studies.aspx , this should help you to make your decision.

Next February 9th 11:30 AM CST I will be delivering a presentation about Troubleshooting ISA Server 2006 Performance issues for Microsoft Partners, if you are a partner and deal with ISA Server you should watch this presentation. Here it is the agenda with the core topics that will be covered:

• What about understanding the issue first?
• Source of the Performance Problems
• Core Components
• Data Gathering
• Troubleshooting Sample Scenarios
• Slow Internet Access
• ISA stops answering requests
• Advanced topics

The registration is open at https://training.partner.microsoft.com/learning/app/management/LMS_ActDetails.aspx?UserMode=0&ActivityId=573031

See you there !!

Consider a scenario where you just added a new application on UAG 2010 portal, you activate the configuration and see the window below:

Right after click in Finish button we tried to access the portal and didn’t see the application in there. After less than a minute we tried again and the new application is still not there. Although the window above says that the configuration was saved, you should read the full message and realize that it says: “Synchronizing changes might take a few minutes”. In order to see if the synchronization is completed you should go to Forefront UAG Activation Monitor, in this case the window that we saw was the one below:

This means that the activation is not completed yet, after some time the activation was completed and the window showed the full log of what it was done during this process as shown below:

Notice that at the end of this log output there is a message saying that the activation completed successfully, at this point we tried to access the portal and the application was there.

The Open Source Web Application Security Project (OWASP) has the Top Ten Project document currently in release candidate that can be downloaded from http://www.owasp.org/index.php/File:OWASP_T10_-_2010_rc1.pdf and the presentation that covers the main changes can be downloaded from http://www.owasp.org/images/a/a1/AppSec_DC_2009_-_OWASP_Top_10_-_2010_rc1.pptx. This document is very interesting from the application security standpoint and emphasizes some practices that are in place for years but are still playing a big role when mitigating today’s threats.

Very worth it to read it.

FWEngmon can be used in many circumstances and here are some great examples on how to use this tool:

http://blogs.technet.com/isablog/archive/2008/03/12/bi-directional-affinity-in-isa-server.aspx

http://blogs.technet.com/isablog/archive/2008/06/24/server-publishing-with-isa-server-2004-2006-and-route-relationship-between-networks.aspx

http://blogs.technet.com/isablog/archive/2007/06/25/rpc-over-http-logging-wildness.aspx

With Forefront TMG 2010 this tool is gone, but no worries, now it is actually much better since is part of the netsh command. Here it is an output of the command that shows the active sessions:

There are much more options available, just use the /? And you will see:

Now go ahead and start playing with this new built in toy.

I know that you are anxious for this book to be out since TMG 2010 is already a reality today, I am too. The good news is that it is close, next month it will be available as announced this week on MS Press Blog. While you are waiting for this book to come out, we got a pretty cool preview for you, two full chapters so you can read it and get the feeling of the core book writing style. Besides those two sample chapters, here it is the final and full open cover for the book:

Access http://blogs.msdn.com/microsoft_press/archive/2010/01/13/forefront-tmg-2010-administrator-s-companion-sample-chapters.aspx for more details about the sample chapters and how to download it.

Here it is something very cool to start this year: TMG Product Group released the hardware recommendations that applies to Forefront TMG 2010. This article also introduces my friend Tom Shinder (now MSFTE) as technical reviewer. I’m sure that from now on much more articles with his name will appear (welcome Tom !!). Read the full article at: http://blogs.technet.com/isablog/archive/2010/01/12/hardware-recommendations-for-forefront-tmg-2010.aspx

Happy TMG Deployment !!

Introduction

This post is about an interesting issue that I recently worked where firewall administrator published an internal web site through ISA Server 2006 and when it tries to load the main page some components didn’t load correctly and the error message below appeared: