Last two months for some reason were pretty busy of calls for ISA Server issues where customers were running on a non supported scenario.  Interesting enough, the articles for non supported configurations are out there since ISA Server 2004. Maybe it is time to refresh your favorites and to start add those articles to it. Here you will find supportability boundaries, limitations and unsupported scenarios:

Article: Troubleshooting Unsupported Configurations

Description: In the above article you will find nice explanations about some behaviors, including the reason why ISA Server does not support multiple default gateways.

Article: Best Practices for Performance in ISA Server 2006

Description:  this article will explain the options to deploy ISA Server 2006 in a virtual environment.

Article: Configuring ISA Server 2004 on a Computer with a Single Network Adapter

Description: this article is also valid for ISA Server 2006 and it has the limitations and unsupported scenarios for ISA Server when running in a single NIC system.

Besides the official ISA Server TechNet Library articles, we (ISA Server Team members) are documenting in the ISA Team Blog behaviors  that are expected. Here are the articles that were published so far:

Understanding By-Design Behavior of ISA Server 2006: Buffering and Streaming Web Publishing Rule Content

Understanding By-Design Behavior of ISA Server 2006: Using Kerberos Authentication for Web Proxy Requests on ISA Server 2006 with NLB

Files larger than 512MB are not served from cache after ISA Server firewall service is restarted

The tip for the IT professionals that are implementing ISA Server 2006 is to review those articles before start any deployment. I know how frustrate it is to build the whole infra-structure and when call to CSS to open a ticket get the bad news that the environment is not supported. Although this can be a frustrated experience, you should feel glad that this product has very known and public supportability boundaries. This helps you to understand what can and what cannot be done before start deploying your ISA Server.

The CSS Security Team released yesterday a very cool tool that combines features from WSUS MPSReports tool and the FCS MPSReports tool.  Click it here to see for more details and download the tool.

1. Introduction

The process flow is very known by all IT professionals: user can’t access a web site and calls the Help Desk. First contact, initial troubleshooting, can’t fix and it calls the network admin. At that point, after troubleshoot his “piece” of the puzzle, it says: well, it got be our firewall that is not allowing that.  

This chain started before I received this call where the user couldn’t open a web site that has some pages in MHT format. The error message was: page cannot be displayed.  This issue has all elements to point ISA as the culprit:

·         Traffic doesn’t work from all workstations behind the ISA.

·         Accessing the web site from the ISA Server itself it works.

2. Narrowing Down

There is a very interesting book about troubleshooting called The Art Of Systematic Troubleshooting by Shawn Pinnock and if you read this book you will understand that one of the key elements of the troubleshooting technique is: ask questions.   This was what I did; we need to know more and the following questions were asked:

·         Does this ever worked?

o   Answer: No.

·         Is this a new ISA Server installation?

o   Answer: Yes.

·         Does the issue happen from IE 6 and IE7?

o   Answer: I don’t know about IE7 we just have IE6.

·         Does the issue happen only for this web site or all web sites?

o   Answer: I don’t know, I just tested from this one..

·         Can you access http://www.yuridiogenes.com.br/mht/primeirolugar.mht ?

o   Answer: Let me try…no, got the same problem.

·         Do you have any other workstation with IE7 that you can try?

o   Answer: no, we don’t have IE7 at all.

3. Gathering Data

If those answers in mind, I decided to move on and get a sample data from the ISA Logging and netmon trace from both ISA interfaces (internal and external). When I got the data I could that things were clear on the ISA Server side and that although everything points to be an ISA issue, at that point I started to believe that it was not. Here is way:

Diagnostic Logging was showing that the data was transferred to the host:

Figure 1 – Diagnostic Logging showing the traffic.

Netmon trace shows the data being transferred to the client:

What seems to be is that ISA transfers all the data and the client doesn’t render the page correctly. Very odd, but definitely it didn’t seems to be an ISA Server issue.  Customer than realized that he could try to repeat the test from the DC, that has Windows Sever 2003 with IE7 (yeah, I know he said he didn't have IE7) and the result was: it worked !!

4. Now What?

If you don’t keep asking most likely you will not have the information that you need. When I saw that a click came in and I asked: did you deploy those workstations through an image or something? The answer was: yes, we did.

Now things start to make more sense, we have a common problem in the OS or IE level. The next approach was to use FileMon and see what we were trying to access when we were open that page and the answer was clear:

As you can see we were missing the inetcomm.dll and this file is used by both Outlook Express and Outlook. Guess what: all the workstation didn’t have Outlooks Express.  Besides that the template workstation was tightly hardening by the security admin.  But what this has to do with MHT? Here it is the missing dot:

What we did was manually copy the inetcomm.dll from another XP SP2 (mine, I actually sent to him) to the System32 folder, re-register the inetcomm.dll. After that we were able to access the web page successfully.

To prove the point that not having the Outlook Express was not really the issue, we did two tests:

·         Got a workstation and uninstalled the Outlook Express from it. The file was still there and we were able to access the system.

·         Renamed the inetcomm.dll  to inetcomm.old and ran the sfc /scannow. The Windows File Protection asked for the CD and replaced the file into the %systemroot%\system32\dllcache, after that we manually copy to %systemroot%\system32. Fixing the issue.

This means that somehow this file was manually removed during the hardening process that was done in the template workstation.

5. Conclusion

I have no doubt that it is really good to see people concern about security, however after hardening a system and deploy to hundreds of users it is VERY important to perform tests. Besides, there are supported ways to hardening a system without compromise the core functionality of it.  For Windows XP, take a look into the Windows XP Security Guide.

Traffic Simulator and the built in Diagnostic Logging capabilities are two new features included in ISA Server 2006 SP1. Today Dr Tom Shinder released the part 2 of the deep dive explanation about those features. I strongly recommend you to read both articles and get the full understanding of how each feature works.

Your New ISA Firewall: ISA 2006 Service Pack 1 (Part 1)

Your New ISA Firewall: ISA 2006 Service Pack 1 - Part 2: Traffic Simulator and Enhanced Diagnostic Logging

1. Introduction

The IAG 2007 allows you to granular control for the URLs that are used for the applications that are published through the portal. Such flexibility and robustness can also cause problems if the administrator doesn’t know exactly how to make the appropriate restrictions. First thing that needs to be done is to understand how the application that you are trying to publish works. Without this understanding you might end up having problems such as the ones that are described in this post.

2. Symptom

The scenario is: customer is publishing the Company’s Payroll Web Site through the portal. When the user accesses the application, the first page has lots of missed images, as show below:

Figure 1 – First problem: missing images.

In a hurry to check the employee’s information the user didn’t care about the images and after typing the employee’s number he clicked in submit. The second error came up as show below:

Figure 2 – Second problem: restriction to access the web site.

3. Moving Further and Understanding the Issue

Those problems might have or not have the same root cause, one single culprit. It is early to say that one single component is causing this issue, thus you need to know where to look first to than start narrowing down the issue. For this case the first thing that we check is IAG Web Monitor, Event Viewer option, Security session:

Figure 3 – Warning ID 67.

This event was triggered because the URL for this application has an illegal path. Time to review the properties for this application. Here are the main tabs that we need to review for this type of error:

Figure 4 – Reviewing the application.

What we have here is:

·         The address is correct.

·         The path in the Web Server’s Tab is correct.

·         The Web Settings are enough for this web site to work.

·         The application’s URL in the Portal Link is also correct.

What will be the next step?

6. Gathering Data

Last week we troubleshoot SSL Handshake using Fiddler, although this is a powerful tool it will not help that much in this case since we can’t see the content because the access from the portal is encrypted (SSL). However, there is another great tool that can show a lot more about the HTTPs traffic; this tool is called HTTP Watch. The HTTP Watch Basic version is free however has limited functionality.  The full version allows you to actually see the code of the page that you are trying to browse and much more details about the communication.  

That will be one option (at least for this case) will be to get a netmon internally while trying to access the web server from the IAG itself.  When I say at least for this case, is because the traffic inside is not encrypted (HTTP only), therefore we can see more details about this request. What we see in this case is:

GET Request and Response for the main page

GET Request and Response for the images that appears in the home page

At this point we know that there are more folders within payrollcontoso virtual directory. For this case, the IIS Administrator was not available and we couldn’t get more details about all the folders that exist within this directory. Knowing that, it is time to review the URL Set configuration in the IAG Trunk.

5. URL Inspection

If you review the Chapter 6 of IAG Advanced Configuration Guide you will understand in details how the content inspection happens when IAG is processing a request. In summary when IAG receives a request it checks if the path for this URL is allowed based on the URL Set and since we have the Check Out of the Box Rules selected in the Web Settings tab, it will also check for legal characters and forbidden encoding. For this particular error most likely we are not facing an out of the box rule restriction, otherwise the error message will be different and the event id will be 48. The URL Set for this case has the following rule:

Figure 5 – URL Set.

Here it is the problem:

Although the action for the Payroll_Rule1 has Accept the URL has the following definition:

The URL settings use Regular Expression (RegEx) that has granular capabilities. The definition above doesn’t cover all the content within /payrollcontoso/, if you are used to ISA Server you will think: hum, in ISA Server this will be enough. Yep, it will, but not in IAG. To cover all the content within /payrollcontoso/ you will need the following syntax:

Since the dot character "." matches any single character and an expression followed by an asterisks "*" can be repeated any number of times including zero, this should be the expression that you want to use to cover all the content within a directory.

After changing this setting and activate the configuration the Payroll page was successfully accessed.

6. Conclusion

By understanding how IAG evaluate and process the URL you will be able to mitigate issues prior to deploy you application through the portal.

1. Introduction

One common problem that is happening these days when customers start to deploy IAG Portal is due one simple fact called “certificate”. There are many mistakes that can happen while implementing a new certificate to assign to an IAG Portal.  This post will explain the symptom, how to identify the problem and how to fix it.

2. Symptom

The scenario was: customer has the main IAG Portal working and he needs to create a new webmail portal for the company. This webmail portal will be accessed from outside by using the name https://webmail.contoso.com. When outside users try to access this web site nothing comes up, the Internet Explorer returns the error message saying: Page Cannot Be Displayed. Here the repro lab for this scenario:

Figure 1 – Scenario used to reproduce the issue.

3. Gathering More Information on the Client Side

When dealing with IAG problems like that, it is important to validate first the following items:

·         There is any edge firewall in front of the IAG?

·         If there is one, is it allowing the traffic to the IAG box?

If it is allowing the traffic, then check if the client is able to establish the SSL Handshake with the IAG Server. To do that you can use the following approach:

·         Install Fiddler and Netmon in the external workstation that you are using to make the connection.

·         Launch Fiddler and Netmon in this workstation.

·         On Netmon, start a new capture.

·         Open Internet Explorer and try to access the URL.

Here the result for this example:

1. On Netmon you will see a pattern like that:

TCP Three Way Handshake happening:

Client tries to initiate the SSL Handshake:

Destination Server Finishes the Connection

2. Using the Fiddler result, you can also look to the following to confirm the behavior:

Figure 2 – Fiddler result.

As you can see above, the client sends the SSL ClientHello to initialize the handshake many times (see left panel). On the right panel (Session Inspector) you will see the details about the ClientHello handshake and the ciphers that are supported by the client. We are expecting the ServerHello that never comes back.

4. Reviewing the IAG Configuration

At this point we know that something is not correctly configured in the IAG Trunk and we also know that it is related to the SSL since we were unable to see the ServerHello from the IAG. Therefore the first thing that needs to be checked is the Certificate.

Open the IAG Configuration Console, click in the Webmail trunk and click in Configure under Advanced Trunk Configuration. In the General tab you will see the window below:

Figure 3 - Certificate for the Webmail Trunk.

Unfortunately this doesn’t means that the certificate is good since the IAG doesn’t do certificate validation (like ISA Server does). But, here you will see at least which certificate you are using on this trunk. After look at this the next step is to open the IIS Manager, right click in the webmail virtual directory choose Properties, Directory Security and then click View Certificate. Here what I have on this case:

Figure 4 – Certificate without the private key.

Here it is the problem, which is a very common mistake. What happened here is that the user just imported the CER file to the personal store, which doesn’t have the private key.  The IAG doesn’t complain about that (like ISA Server does), it accepts and you might think that everything is good at that point.

5. How to Fix It

The first step to fix is to remove that certificate from the IIS Virtual Directory by following the steps below:

1. Open the IIS Manager.

2. Right click in the virtual directory that has the wrong certificate (in this case webmail), choose Properties.

3. Click in Directory Security.

4. Click in Server Certificate.

5. Click in Next in the first window and choose the option Remove the Current Certificate and click Next.

6. Click again Next until you have the option Finish.

Also, make sure that the certificate is deleted from the local store. To do that, follow the steps below:

1. Click Start, Run and type MMC.

2. Click in File, Add Remove Snap In and click in Add.

3. Choose Certificates and click Add.

4. Choose Computer Account, click Next (leave the default option) and click Finish.

5. Click Close and click OK.

6. Expand Certificates, Personal and Certificates.

7. Right click in the certificate that you excluded from IIS and choose Delete.

8. Confirm saying Yes.

Now you need to obtain the certificate that has the private key, the file will have the PFX extension. If you are using Windows Certificate Services, make sure that the following option is enabled when you are exporting the certificate:

Figure 5 – Exporting the certificate with the private key.

After export the certificate with the private key, import it again in the IAG box. After import the certificate on the Personal Store of the IAG Server, open the IAG Configuration, make sure that under the advanced configuration trunk you have the certificate there (should be same as figure 3) and activate the configuration. This activation will allow the IIS commit the changes and bind the new certificate to the virtual directory. Now, if you open the IIS Manager, the certificate will appear like this:

Figure 6 – Correct Certificate.

6. Testing the Access to the Portal

If you use the same tools described in the session 3 while opening the web site you will see the following result:

1. On Netmon you will see a pattern like that:

TCP Three Way Handshake happening:

SSL Handshake:

2. Using the Fiddler the result may vary because of the certificate, but the core part is that we have an answer from the server now:

Figure 7 – Now we can see the answer from the IAG Server.

Above you have an error represented by the red circle because my external workstation doesn’t trust the certificate authority that issues the webmail.contoso.com certificate. If you look to the above figure, you will see that the right panel shows the attempt to get the certificate for dallas.cotnoso.com, which is the CA that issued this certificate.

When the workstation doesn’t trust the CA that issued the certificate a pop up window like the one below appears warning about that:

Figure 8 – Security alert.

7. Conclusion

The simple fact that you don’t import the correct certificate can cause this entire headache. This article covered the identification of the problem, the understanding why this happen and how to fix it. I hope you’ve enjoyed.

Last July 8th Microsoft released the security update MS08-039 for OWA, the following Exchange versions are affected:

If you question is: can ISA Server 2006 help to mitigate this attack? The answer is that it potentially can since ISA Server 2006 can block cross site scripting by inspecting the HTTP requests and identifying commands and tags that are common in server responses but are not common in client requests. For more information about this review on ISA Server TechNet Library the problem and the solution.

I'm really glad that after two weeks of long work writing, reviewing, testing and diving into this project the result pays off. The list of IAG KBs is growing at support.microsoft.com as a result of this huge team effort that we have here at Microsoft. It was an amazing work done from all team members.

Here the first wave of 18 KBs and more will come soon, stay tune at support.microsoft.com: 

This post is about a case that I recently worked here in CSS. The problem was that customer has an ISA Server 2004 publishing OWA from Exchange 2007 and he was getting the error below:

500 Internet Server Error – The target principal name is incorrect

Everything was running just fine until they changed the certificate in both places (ISA and Exchange). The certificate used to be mail.contoso.com and it was changed to *.contoso.com.  After this change this issue started to happen when it try to access the OWA from outside. Looks familiar? I guess so right, this is known issue explained here:

The reason why I’m bring this up now is because people are starting to renew some certificates and are planning on what path to choose as far as certificate type is concern. Wildcard certificate is good, but be aware of the above limitation on ISA and also that Exchange 2007 has some limitations on that too, such as this one below:

Wildcard Certificate Causes Client Connectivity Issues for Outlook Anywhere

http://technet.microsoft.com/en-us/library/cc535023(EXCHG.80).aspx

Note: Exchange Team strongly recommends using SAN Certificate and now with ISA Server 2006 SP1 being able to read the SAN Certificate we have the perfect match.

For more info on Certificate for Exchange read this cool article on the Exchange Team Blog site:

http://msexchangeteam.com/archive/2007/07/02/445698.aspx

1. Introduction

Last week I was helping out a friend in our team during a support call where the scenario was really unusual. The topology was similar to the one below:

Figure 1 – Network topology for this sample scenario.

Customer was using two ISA Servers 2006 Standard Edition Single NIC to publish the OWA 2007. He was using a L5 based switch to balance the traffic among the servers.  When I say that the scenario is unusual is because on this case customer was using two ISA Servers 2006 Standard Edition to publish this resource.  This means that he have to create rules on both ISA Servers, in other words: to guarantee that everything was correctly configured he needs to carefully configure the same rules on both, manually. High cost administrative task right there!

2. Problem 1 – The Cookie

The first problem that customer was dealing with was because he was using the following option on ISA Server:

Figure 2 – Cookie based load balance.

The problem with this option was caused by the switch that was not keeping the cookie and therefore the communication was getting lost, we changed to Source-ip based and started to work. Later on, customer fixed the switch issue using the recommendations from the vendor.

3. Problem 2 – The Verifier

The other problem was the intermittence while accessing the OWA page, sometimes it was working, sometimes was not. After closely analyze the netmon trace we identify that the traffic was always going to one particular CAS when it was working and when it was not working it was going to the other CAS. We narrow it down the problem to be on the connectivity verify farm in one of the ISA Servers.

Figure 3 – Connectivity Verify.

Since customer has to do the publishing in each ISA Server separately, we have a mistake on the connectivity verifier for the ISA Server 2. After fix that everything started to work just fine. But some lessons were learned here.

4. Conclusion

After six hours call the lessons learned (mainly for the customer) were:

·         To align high availability and low administrative cost in a solution with ISA Server make sure to use ISA Server 2006 Enterprise.

·         Look out of the box; do not think that the issue is only on ISA Server. In this case, the first problem was caused by the switch.

Kind of short scenario, but I thought will be good to share to make sure that if you are going to implement something similar, that you know what some of the catches are.

The ISA Server 2006 SP1 was release today (couple of minutes ago) and you can download from the link below: http://www.microsoft.com/downloads/details.aspx?FamilyId=D2FECA6D-81D7-430A-9B2D-B070A5F6AE50&displaylang=en

We do have some good articles out there already that talks about the functionalities of this release, review some of them:

Jim Harrison reveals ISA Server 2006 SP1

ISA Server 2006 SP1 Demos by Yuri Diogenes

Your New ISA Firewall: ISA 2006 Service Pack 1 (Part 1) by Tom Shinder

ISA Server 2006 Service Pack 1: New features and enhancements by Marc Grote

The endpoint drive mapping feature on IAG is a functionality that a lots of administrators wants to use when configuring the IAG Portal due the transparency to the end user. However this functionality still not supported for Windows Vista users, even with IAG 2007 SP1. For more information on this read the IAG 2007 SP1 release notes:

http://www.microsoft.com/technet/forefront/edgesecurity/iag/releasenotes.mspx

There are no workarounds or fixes available for drive mapping on Vista to resolve this issue.  As a possible alternative you can use the File Access functionality to expose the drive via the IAG File Access Web interface.