1. Introduction

This post is about a very interesting case that I worked this week, the initial scenario was:

·         ISA Server 2004 Publishing OWA

·         FBA Enabled on Exchange Server

·         ISA Server using RSASecurID as authentication on the Web Listener

On this particular scenario we were doing “two factor authentication”, first using RSASecurID and second using FBA on CAS as explained below:

Figure 1 – How the scenario was setup

This whole scenario was working fine until they change from ISA Server 2004 to ISA Server 2006.

2. The Issue

After replacing the ISA Server 2004 to ISA Server 2006 the following behavior was notice:

·         If we try to access https://mail.contoso.com/owa it works just fine. We had the RSA screen first and then the OWA FBA Page that was coming from CAS Server.

·         If we try to access https://mail.contoso.com/exchange  we got the RSA page, authenticate but then instead of receiving the OWA FBA page we got a blank page. Internet Explorer appears to be processing something in background but never opens the page.

The reason why we needed to use the “/exchange” in the OWA URL is because there is still some user’s mailbox that were residing on Exchange 2000 Back End Server. For backward compatibility Exchange 2007 keeps /exchange, /public and /exchweb virtual directories to allow users in this scenario to access their mailbox through OWA. When you have FBA enabled on the /exchange folder, what happens is that the request will be redirect to /exchweb since it is in there that the forms reside.

Interesting facts:

·         ISA Monitoring / Logging not showing any error or deny, all the communication was green and flowing normally.

·         If we try to connect from inside (bypassing ISA) it works fine.

·         If we change the authentication method on the /exchange folder for basic instead of FBA it works. In this case the second authentication (after the RSA) is a prompt (since is basic) so the user can type his credential.

With that we had the following component as main suspicious at that time:

·         Folders /exchange and /exchweb on CAS Server were having some type of issue when the traffic was coming from outside (passing through ISA) since internally was working fine.

At that point I engaged an Exchange Team to validate the settings for those virtual directories.  My friend Vandy Rodrigues, from the Exchange CSI Team started to review that.

3. The Eternal Loop

After reviewing the whole configuration, validated all the settings, permissions and everything he told me: Yuri, all the Exchange settings are clear, no errors. What should I say to him at that point? Nothing more than: Roger that buddy!

Moving further in the collaboration we got a netmon trace in the CAS Server while client was trying to perform the logon to https://mail.contoso.com/exchange. The result was a beautiful eternal loop, check this out:

ISA Sends the GET Request to the CAS Server:

The CAS Server sends the 302 Redirects saying that the request needs to go to /exchweb:

…and this GET and Redirect keeps going on and on forever:

4. The Resolution

After researching more and more I found that the conditions for this problem were perfectly matching with KB935206. Not only the conditions but also the symptoms were similar. The solution was exactly that! Read KB935206 to better understand the resolution (and the cause) before apply it and notice that the hotfix itself doesn’t fix the problem; you have to run the script to make it happen.

Microsoft Malware Protection Center released last Friday an update about the new Conficker variant, as MMPC’s blog says: “The new sample has modifications which introduce new backdoor functionality. Previous versions of Conficker patched netapi32.dll in memory to prevent further exploitation of the vulnerability addressed by bulletin MS08-067.”

Check it out the complete post at http://blogs.technet.com/mmpc/archive/2009/02/20/updated-conficker-functionality.aspx

Is not only ISA fans that are excited about TMG, marketing is already giving great feedback about this Beta, check it out some of them:

Microsoft's TMG adds antimalware, SSL inspection
CNET

Microsoft Enhances Security Products
InternetNews.com

Microsoft Forefront Threat Management Gateway Beta 2
BetaNews

Microsoft's TMG adds antimalware, SSL inspection
ITvoir

Almost one year passed since we first started this project, from the book conception, TOC, proposal until start writing the first line in Chapter 1. We are now hitting 50% of the book and the sensation is that this guy is going to be really good. To celebrate this 50% mark we just release our Book’s web site that will have more info about it, is called http://www.mstmgbook.org. The road is long, but definitely worth and totally amazes to work with those bright minds that are part of this book.

Introduction

First let’s understand what silent quits means:

Note: Although this article is for Exchange these functions are Windows (Win32) related.

What about graceful shutdown, what is that? That’s simple: a service received an expected command to gracefully stop.

The Scenario

The scenario of this article was based on a real case where customer had to manually start Firewall Service every day, it was “apparently” quitting every night.  The problem with a silent quitting is that debugger will not catch; therefore there will be no dump file to analyze. Even knowing that we tried to get a dump and of course the result was a 1st chance exception dump, no second chance. Therefore we got useless data.

Moving Forward

After researching more and more we found out that Telephony Service was set to disable and ISA Server Control depends on Remote Access Connection Manager that depends on Telephony Service:

Figure 1 – ISA Server Control Dependencies.

Looking the System Log, there following sequence of events were showing up:

In the application log we got the prove that this was not a silent exit, it was actually a graceful shutdown:

Now What?

If those services are stopping every night and the administrator needs to manually start those, this leads to a conclusion that something (a process) is stopping it. For a domain joined ISA the first thing you shoul check is Group Policy. A simple thing that can be done without impact the production just to check if ISA Server is receiving any policy is run the command RSOP.MSC. The result for this case was shown in Figure 2:

Figure 2 – RSOP.MSC result.

Bingo !!! Now everything makes sense. What was happen here was that ISA Server was inside of an OU that has a policy which was disabling those services. To fix that we created a new OU, moved ISA Server to this new OU and block inheritance in this OU.

Conclusion

Sometimes IT administrators using their best of intention disable some services that are considered not necessary from a Windows perspective (attempting to hardening). However, for ISA Server this needs to be carefully done since it can stop Firewall Service which will cause downtime in your Internet access. Before do this, review the article below that has a list of services that ISA Server depends on:

http://technet.microsoft.com/en-us/library/cc302488.aspx

Here it come another post about Firewall Service crashing and again the best approach to catch this crash was to attach a debugger to the firewall service (see this article to know how to do this).  Result was quiet interesting because at this time there was no third party involved. After some research I found out that this was a known issue and fixed by KB http://support.microsoft.com/kb/956268.

This brings another important point: keep your ISA Server up to date on patches. Don’t think that because you already have ISA Server 2006 SP1 that you are on the latest bit version for ISA. Keep watching for new updates, mainly for “hotfix package”. In this case instead of applying 956268, I applied the latest “hotfix package”, which was http://support.microsoft.com/kb/960148. Since the hotfix is cumulative, this package not only fixes the crash issue but also other issues that might happen, such as:

Notice that this is big set of issues that were fixed and therefore you shouldn’t ignore. In summary, the tip of the day is: keep your ISA Server 2006 up to date with the latest hotfix package.

Yesterday I was talking with Richard Hicks and he was telling me how excited he was with the new set of features that TMG has.  This is indeed true, many people got really excited with the package of new features that TMG Beta 2 brought, but there is something that you might not realize it yet. TMG is design to run in a Windows Server 2008 64-bit platform…ok, and what this means? Well, this means robustness to your edge device. This means the capability of scale your environment to other levels while your edge device is not the bottleneck of the traffic. To summarize the technical values of what this really means, check it out this table:

Source: http://support.microsoft.com/kb/294418

If you think about it is very interesting to see this in another perspective, because for a firewall standpoint it is not every day that you see such high numbers.

Last Wednesday I delivered a presentation here at TechReady 8 (in Seattle) about TMG Beta 2 SMTP Protection (which is a feature that I really like since I worked with Exchange for so many years) and today I’m really happy to share with you that Forefront TMG Beta 2 is available for public download at Microsoft web site. This TMG version has so many news that you have to take some time to install, analyze the new GUI, read the deployment guide and see with your own eyes the exciting new set of features that this version offers. Let me highlight some of them:

• SMTP Protection

• Network Inspection System

• Integration with the Forefront Security Suite (codename Stirling). For this you will need Stirling Beta 2.

Now go ahead and download TMG Beta 2 and enjoy this great release. On the page below you will find the bits, release note and deployment guide. So you are all set to test, evaluate and send feedback about TMG Beta 2:

http://www.microsoft.com/downloads/details.aspx?FamilyID=e05aecbc-d0eb-4e0f-a5db-8f236995bccd&DisplayLang=en

1. Introduction

This post is about a scenario where users were not able to authenticate on a FBA page published by ISA Server using LDAP as authentication repository. The error message that was showing up there was:

Figure 1 – Unable to authenticate.

Although it says to double check the domain name or password to see if it is wrong, this is a generic logon error and this may not be the case. We recently wrote an article at Tales from the Edge that has a troubleshoot framework for LDAPs authentication on ISA. More info check the article at http://technet.microsoft.com/en-us/library/dd316279.aspx.

2. Logging is your Friend

The ISA Server realtime logging can be very helpful in scenarios like this. In this case the error message was the one below:

Figure 2 – Error 58.

As you can see in the figure above the error message says that it was not possible to perform the requested operation. This can be a good start, but you can see even more information if you copy the whole logging to the clipboard using the option below in the task pane:

Figure 3 – Using the Clipboard option.

After copy, paste in a notepad file and save as TXT. Best thing to do is to open this file in Excel to see all the fields and be able to filter. After opening the file in Excel, I was able to see a key error in there:

Figure 4 – Using Excel to filter the logs.

Notice that in the Authentication Server field it says: dccont\No server available. This is it!! Now we can conclude that:

·         ISA cannot reach the DC for some reason:

o   Networking issue?

o   Name resolution issue?

o   DC not answering?

Before go crazy and start to investigate this deeper, what about just try to ping the server that are in the LDAP Server Set? This is what I did and the result was:

Figure 5 – unable to resolve the name.

Bingo, unable to resolve the name. After fix the name resolution problem the issue was gone and the authentication worked.