ISA Server Product Team Blog

Tuesday, April 29, 2008 12:47 AM

Troubleshooting OWA 2007 Publishing Rules on ISA Server 2006

1. Introduction

Sometimes troubleshooting publishing rules can be really challenging. The main reason for that is because of the thin line between the published server and the edge firewall (in this case ISA Server). Where is it broken? Is my ISA Server not allowing the connection from outside or my published server that is not answering the request from ISA Server?

The sequence of articles written by Jim Harrison about RPC over HTTP helped to clarify how the protocols works and also to troubleshoot the main issues that happen when publishing Outlook Anywhere through ISA Server 2006.

This post was actually born internally after many collaboration calls with Exchange Engineers and narrowing down if it was an ISA Server problem or an Exchange problem. We could identify the most commons scenarios of errors while publishing OWA through ISA Server 2006.

2. General Recommendations

The main recommendation is to use the official white paper that is a step by step configuration guide to OWA 2007 through ISA Server 2006.

2.1. Authentication Options

When you are publishing OWA 2007 through ISA Server 2006 you have two options for configuring FBA (Forms Based Authentication):

· Option 1 (Most Common): Disable Forms Base Authentication on the Exchange OWA side and enable FBA on the ISA Server 2006 Web Listener.

Figure 1 – Authentication properties.

Note: When you have Basic Authentication selected on the OWA as showed above, make sure that the Authentication Delegation tab on the ISA Server 2006 OWA Publishing rule has the Basic Authentication selected.

· Option 2 (Rarely Used): Do not authenticate on the ISA Server 2006 and use the Forms Based Authentication on the Exchange side:

Figure 2 – Second option for authentication.

2.2. Certificate Considerations

When you are publishing OWA 2007 you can still use SAN Certificate on the Exchange OWA side. However the FQDN name that appears on the To Tab on the ISA Server 2006 OWA Publishing rule needs to match with the first name on the SAN Certificate as show below:

Figure 3 – SAN Certificate details.

Note: Other queries for the other names that are available on the SAN will not work since this version of the ISA Server 2006 (with Supportability Update on it) only uses the first name on the SAN Certificate. This feature is planned to be available on ISA Server 2006 SP1. We had a demo in early April/2008 about ISA Server 2006 SP1 in TechEd Eilat (Israel) and the initial plan is to release SP1 in late summer 2008 but issues could cause delay, so stay tune on www.microsoft.com/isaserver .

For more information on SAN certification and Exchange Publishing rule read the blog Certificates with Multiple SAN Entries May Break ISA Server Web Publishing on Microsoft ISA Server Blog Team.

3. Troubleshooting the Most Common Scenarios

3.1. I can access the OWA page, authenticate using Forms Base but after the first authentication I receive another page asking for authentication again.

Make sure that you have forms base authentication enabled only on ISA Server or only on the Exchange. Review the session 2.1 for more information.

3.2. After authenticate on the OWA Forms Base Authentication I receive the error: Target principal name is incorrect.

Make sure that of the following:

· The name on the certificate that is installed on the Exchange Server matches with the name on the To tab for the OWA Publishing Rule on ISA Server. Review session 2.3 for more information.

· The Exchange Server name (usually the CAS Server) is correct on the To tab on the OWA Publishing Rule on ISA Server.

· ISA Server 2006 can correctly resolve this FQDN for the correct IP.

3.3. I cannot even access the OWA Page, when I type the address https://mail.contoso.com on IE I receive the error: 403 Access Forbidden.

When you create an OWA 2007 Publishing Rule only the following paths are allowed by default:

Figure 4 – Default paths for OWA 2007.

When you type https://mail.contoso.com without use /owa or the other paths that are allowed by this rule it is expected to receive the 403 since ISA Server is restricting the paths for security purpose.

3.4. I’m typing the address https://mail.contoso.com/owa on Internet Explorer, I can reach the FBA page, authenticate but after that I’m receiving the error: 404 Not Found.

Make sure that the ISA Server 2006 can reach the Exchange Server that it is using on the To tab. Try to validate using the following methods:

· From the ISA Server ping the FQDN name of the Exchange Server that is in use on the OWA Publishing to make sure that is resolving to the correct IP.

· Telnet from ISA Server to the FQDN of the Exchange Server and use the port that appears on the Bridging tab as show below and see if you receive an answer (telnet exc2007.contoso.msft 443):

Figure 5 – Bridging Tab.

· If all tests succeed then enable logging on ISA Server (ISA Console / Monitoring / Logging) and create a filter where the source is the External network. Try to reproduce the problem and check if you see the error below:

Figure 6 – Failed connection attempt to the Exchange Server (error 10060).

The error 10060 showed above means that the connection timed out. Usually this error means:

· Connectivity issue between ISA Server and the Exchange Server that is being used by this publishing rule.

· The ISA Server is sending the package by for some reason the Exchange Server is not replying in appropriate manner.

Review the following:

· There is any other firewall in between ISA Server and the Exchange Server? If it does, can we bypass that? If we can’t bypass, make sure that the ports are allowed in both ways. Use PortQuery to test that from the ISA to the Exchange Server.

· A netmon trace on the internal NIC of the ISA and on the Exchange Server might show what is happening.

· The IIS Logs on the OWA also will help to see if the traffic is hitting the OWA Server.

3.5. I’m typing the address https://mail.contoso.com/owa on Internet Explorer, I can reach the FBA page, authenticate but after that I’m receiving the error: 10061 Connection Refused.

The error code 10061 means: A connection was refused by the destination host. This error means that ISA Server tried to contact the published server (in this case the Exchange Server) and this server refused the request for some reason.

This usually happens because ISA Server tried to contact the published server on a port that is not allowed on the destination server. Review the Bridging tab on the OWA Publishing rule to make sure that this port matches with the port that OWA is using on the IIS web site:

Figure 7 – IIS properties matching with the Bridging tab.

If you also use the ISA Server Monitoring Logging you will see the error below:

Figure 8 – Failed connection attempt to the Exchange Server (error 10061).

For more information about the error code see the ISA Server 2006 Logging Fields and Values article.

3.6. I can authenticate on the FBA page, I can see my OWA inbox but I’m unable to create a new message and the buttons on the toolbar doesn’t work. Everything works fine internally (bypassing the ISA Server).

Don’t set your mind to think that if works internally and externally doesn’t then the issue is on the ISA Server. There are some 3^rd Party ISAPI filters (for example the one mentioned on KB 924228 ) that could be installed on the Exchange Server (Backend, FrontEnd, CAS or Mailbox Server) that can cause this behavior when the traffic crosses the ISA Server. A more in depth investigation will be necessary to determine the root cause in this situation.

4. Conclusion

As you can see these are the most common problems that might happen during the OWA publishing through ISA Server 2006. If you still have problem, a good advice before call Microsoft to open an incident is to use the ISA Server BPA and check if there is any recommendation on that. ISABPA does a pretty good job identifying mistakes o the configuration for the publishing rule and making suggestions on how to fix it.

Yuri Diogenes

Security Support Engineer – ISA Server Team

Microsoft Texas

Posted by isablog | 0 Comments

Thursday, April 17, 2008 2:32 AM

ISA Server 2006 form base authentication problem using UPN logon format on a multiple domain environment

1. Understanding the Environment

The scenario that I’m going to describe on this post was based on a case where the customer’s need was really specific and the configuration of the ISA Server was restricted by the company’s hardening policy. The topology was similar to the one below:

Figure 1 – Lab used to simulate this behavior.

As you can see above the ISA Server 2006 was a member of one of the child domains and it was using FBA (Forms Base Authentication) to publish an internal web server. The goal was to be able to use UPN format to logon to the Dallas.contoso.com child domain from outside. In this case customer was using multiple UPN suffixes for the same domain.

For more information about multiple UPN suffixes concept review the KB243280.

When the external client uses the SAM format (domain\user) it worked fine no matter what the domain was. However, using the UPN format the authentication was only working for users located on the same child domain as the ISA Server 2006. The error message that was showing up on the Internet Explorer right after the client logon was:

"The server requires authorization to fulfill the request. Access to the Web server is denied. Contact the server administrator.”

2. Reviewing the Data

To better understand the logon attempts we enabled the netlogon logging on the domain controller. To do this run the following command on the Domain Controller:

nltest /dbflag:0x2080ffff

2.1. Reviewing the Netlogon Log and the Security Logs

On the Domain Controller located on the Austin.contoso.com we could see the logon attempt:

[CRITICAL] NlPrintRpcDebug: Couldn't get EEInfo for I_NetLogonSamLogonEx: 1761 (may be legitimate for 0xc0000064)

11/30 08:45:50 [LOGON] SamLogon: Network logon of Austin.contoso.com\bob@dallas.contoso.com from ISASRV1 Returns 0xC0000064

The error code 0xC0000064 means that the user does not exist and we can confirm that on the security log of the ISA Server:

Event Type: Failure Audit

Event Source: Security

Event Category: Account Logon

Event ID: 680

Date: 04/10/2008

Time: 11:51:03 AP

User: NT AUTHORITY\SYSTEM

Computer: ISASRV1

Description:

Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0

Logon account: bob@dallas.contoso.com

Source Workstation: ISASRV1

Error Code: 0xC0000064

3. Hardening

On the first paragraph of this post I mentioned that the ISA Server was hardening due the company’s security policy. One of the changes that it was implemented on this case was the change for the LmCompatibilityLevel entry for 5. During the course of the troubleshooting for this case it was also identified that the issue didn’t happen using the default LmCompatibilityLevel value. But, we couldn’t implement this workaround due the compliance of this server to the company’s policy.

4. The Solution

This is a known issue that was recently fixed with the hotfix mentioned on the KB article below:

Authentication of trusted users fails on a Windows Server 2003-based server if the UPN format is used and if the value of the LmCompatibilityLevel entry is equal to or larger than 3

http://support.microsoft.com/?id=947861

Notice that even if you have Windows Server 2003 SP2 you still need to update the system using the fix above. Also, it is important to emphasize that although the article focus on FBA authentication, the same behavior can occur using normal HTTP authentication. In any of those instances, the SAM format logon is handled the same by ISA Server when calling LSALogonUser API. For more information on NTLM user authentication, review the KB102716.

Yuri Diogenes

Security Support Engineer – ISA Server Team

Microsoft Texas

Posted by isablog | 1 Comments

Filed under: Troubleshooting, ISA

Wednesday, April 09, 2008 8:53 AM

Introducing a New Era for ISA Server

As my first (of hopefully many) contributions to the ISA Server team blog, I want to first introduce myself to the community. My name is David B. Cross and I am the new Product Unit Manager for the ISA Server engineering organization. For many of the community who do not know me, I have spent my last 8 years in the Windows Security organization of the company working in several notable areas such as PKI, smartcards and Kerberos authentication. If anyone is interested, Jeff Jones posted an interview with me some time back: http://blogs.technet.com/security/archive/2007/02/12/profiles-in-security-david-b-cross.aspx.

Now – on to why I am blogging today…Why is this a new era for ISA Server? We have been very busy for the past year and we also have been fairly quiet on the messaging front of our plans. Well, the time is over, and I am excited to say that we can announce what we have been up to! Today we (publicly) announced our next-generation network security product, the Forefront Threat Management Gateway (TMG), a comprehensive network protection solution. Forefront TMG is the future version of the Microsoft Internet Security & Acceleration Server (ISA Server) and will extend the capabilities of ISA Server with new features and security technologies. I know many of you have loved and embraced the ISA Server name and brand for a long time – but it is time for new naming, new logos, blogs, books and of course new technology directions.

Forefront TMG will be available as both a standalone solution but also part of new integrated suites to be released in the future such as the Microsoft Windows Essential Business Server, the recently announced server solution designed for mid-sized companies due out later this year as well as the Forefront “Stirling” suite announced today, a unified protection solution that combines Forefront client, server, and edge security solutions with a single management and policy layer. A “first look” preview of Stirling Beta 1 was shown at RSA this week. It is also available for download.

Why am I so excited about this announcement? To begin, it is the first version of ISA Server that will fully support the Windows Vista and Server 2008 platforms. It also will natively support the 64-bit Windows Server platform which provides significant scalability and security capabilities to the Threat Management Gateway. The three other main areas of enhancement we are announcing today are the following:

1. Multiple Threat Protection: We will enable numerous new protection technologies and capabilities, including integration of the Microsoft Anti-Virus Engine for protection against Internet-based malware and other threats. As part of Stirling, Forefront TMG will also include the “Dynamic Response” functionality to enable shared intelligence and response. This is a major step forward in how our customers rely upon the Microsoft gateway for protection and access to the Internet.

2. Simplified Management: Forefront TMG will include new set-up wizards, improved management interface and enhanced reporting. As part of the Stirling suite, Forefront TMG will be part of the Stirling central visibility dashboard and policy control.

3. Secure Connectivity: Forefront TMG will build on current ISA Server capability around secure Internet access and other connectivity features.

More details about the features in Forefront TMG will be available with the public beta scheduled for the second half of 2008. I wish I could share more details and plans now, but that will come with time. I promise to personally keep you updated as our plans and product evolve to keep you updated here. We really look forward to your feedback on our plans and our first beta. I am sure you will be as excited as we are in finally announcing this next generation of our network security product line. Stay tuned to this channel!

David B. Cross

Product Unit Manager

Posted by isablog | 14 Comments

Wednesday, April 09, 2008 12:55 AM

Understanding By-Design Behavior of ISA Server 2006: Buffering and Streaming Web Publishing Rule Content

Introduction

There are some customer issues that are caused because of ISA Server design or standard behavior that is not clearly understood. This article describes by-design behavior that occurs when you are using a Web publishing rule to serve content to external clients.

By-design behavior

By design, ISA Server 2006 buffers content published by a Web publishing rule and then sends it back to the client. This is shown by the following diagram.

Figure 1 – Content buffered and served by a Web publishing rule.

Streaming or Buffering Content

In some circumstances, you may want to enable ISA Server 2006 to stream the content published by a Web server instead of buffering it. This can be controlled by the application hosted on the Web server, as follows:

· If the application uses a Content-length header specifying the size of the message body, then ISA Server 2006 needs to validate the content against the specified length. In this scenario content will be buffered as illustrated in Figure 1. This is expected behavior.

· If the application uses headers with transfer-encoding: chunked (see section 14.40 of RFC 2068), then ISA Server 2006 can stream the content to the requesting client as each chunk is received.

Note: To enable chunked transfer encoding on IIS see Microsoft Knowledge Base article 278998.

Caveat

When you create a Web publishing rule in ISA Server 2006, link translation is enabled by default. With link translation enabled, when ISA Server 2006 receives a response from the Web server it has to process rule translation as packets are received, and send them back to the client. If ISA Server does not have enough data to buffer and translate, the data is held until there is enough to process the URL translation. The content is send it back to the client in chunks. Even using headers with transfer-encoding: chunked, ISA Server 2006 buffers the content and then sends it back to the client. This is expected behavior. The same behavior is expected when you have compression enabled on Web Listener that you are using to publish the server on ISA Server 2006.

To change this behavior you can disable link translation on the Web publishing rule. This ensures that ISA Server 2006 streams the page if the header has transfer-encoding: chunked, and transfers the chunks to the client as received by the Web Server. It is important to emphasize that not all Web Publishing rules need link translation enabled. For more information on link translation, see Link Translation Concepts in ISA Server 2006 at Microsoft TechNet.

To disable compression for a particular listener, go to the General Page, click in Define HTTP Compression Preferences, click in the Return Compressed Data and remove the listener from there.

Yuri Diogenes

Security Support Engineer – ISA Server Team – Microsoft Texas

Rayne Wiselman

Writer – ISA Server User Education Team – Microsoft Haifa

Posted by isablog | 0 Comments

Filed under: ISA

Thursday, April 03, 2008 1:43 PM

IE 7.0 may fail to open FTP sites through ISA Server Web Proxy

Consider the scenario:

1. IE 7.0 is configured as a Web Proxy client.

2. IE 7.0 accesses an FTP site through ISA Server 2004 or ISA Server 2006 Web Proxy.

3. The FTP url contains user credentials to authenticate to the FTP site.

Under this scenario, the FTP request may fail if the credentials contain characters which need encoding. For example, the less-than sign (<) is encoded as %3c.

IE 7.0 may use double encoding, thus converting the less-than sign to %253c. ISA Server cannot handle this encoding scheme when authenticating to a FTP server.

There are three different workarounds which can be applied to solve this problem:

1. Do not use special characters in the FTP username or password that are double encoded by IE 7.0.

2. Access FTP from SecureNAT or Firewall Client instead of through Web Proxy.

3. Install the IE 7.0 hotfix from http://support.microsoft.com/kb/932562/en-us . With this hotfix IE 7.0 will not use double encode.

Thanks to Steve Frode Nabseth for helping with this.

Doron Juster

ISA Server sustained engineering group.

Posted by isablog | 0 Comments

Wednesday, March 12, 2008 6:53 AM

Bi-Directional Affinity in ISA Server

The reason for having a network load balancing (NLB) Bi-Directional Affinity (BDA) feature in ISA Server is that it guarantees that traffic is handled in both directions by the same array server. This is very important and required if you have a Route Relationship (instead of NAT) between two ISA Server networks (for example between the DMZ and the External network) or if you use a Publishing Rule (Server Publishing or Web Publishing) that is configured with “Requests appear to come from the original client” (this is the default setting for a Server Publishing rule) instead of “Requests appear to come from the ISA Server computer” (this is the default setting for a Web Publishing rule). If for example the BDA feature was not available or for some reason not correctly configured in ISA Server, then you could potentially see that incoming traffic from an external client was handled by one array member, while the return traffic from an internal published server (that is configured to route traffic back out through ISA’s internal Virtual NLB IP as it typically will and should be) back to that external client was handled by another array member. This would cause the connection to fail as there is no connection sharing between ISA array members. Because of this we need to guarantee that traffic is handled in both directions by the same array server, and this is what BDA does for us when we have a Route Relationship between two ISA Server networks or if we use “Requests appear to come from the original client” in a Publishing rule.

Note that when using a Route Relationship between two ISA Server networks or using “Requests appear to come from the original client” in a Publishing rule, then you should make sure that you enable Integrated NLB on both the Source and Destination network interface for the involved networks. Otherwise BDA will likely fail and you may see strange symptoms. If for example you only enable NLB on one network when having a Route Relationship between two networks, then you should see a “NLB Inconsistent Configuration Detected” ISA Server alert being triggered similar to the following:

NLB Inconsistent Configuration Detected

An inconsistency in the Network Load Balancing (NLB) configuration may result in inconsistent handling of traffic between the External network and the DMZ network. When a network rule specifying a route relationship is defined between two networks, NLB must be enabled (or disabled) on both networks. To enable NLB for IPSec remote site networks, enable NLB on the network containing the local tunnel endpoint. To enable NLB for VPN site-to-site and VPN client networks, enable NLB on the selected access networks. Alternatively, for the VPN Client network, you can designate a router for routing traffic according to the static address pool.

Using Integrated NLB in ISA Server means that only the NLB single affinity scheme is supported to distribute the load among the array servers. This is a BDA requirement. Single affinity basically means that only a single IP address (source or destination) is used to make load distribution decisions. ISA Server uses NLB Hook Rules to make decisions either it will use the source IP address or destination IP address as affinity, and you can list these NLB Hook Rules by running “FWEngMon /QueryNLB” from a Command Prompt on one of the ISA Servers in the array. That we hash the affinity based on the source IP address for incoming traffic and based on the destination IP address for the return traffic (or vice versa) is actually how BDA works and what guarantees that traffic is handled in both directions by the same array server.

FWEngMon.exe is available both for ISA Server 2006 and ISA Server 2004 and can be downloaded here:

http://www.microsoft.com/downloads/details.aspx?familyid=01fc5551-5d44-4a99-966a-bd86caeb43d7&displaylang=en

http://www.microsoft.com/downloads/details.aspx?familyid=f3306399-d4f9-4989-865e-c61f8293c330&displaylang=en

When using Integrated NLB, ISA Server works with Windows NLB to automatically configure Bi-Directional Affinity (BDA). Therefore, you as an ISA Server administrator typically do not have to do any additional steps. You just have to make sure you enable Integrated NLB on the ISA Server network interfaces in ISA Management Console and BDA will work by default and by itself. However there’s always exceptions and you may still experience unexpected behavior as you will see below.

Let us take the following scenario to try and explain the above NLB Hook Rules further. Let us say you run two ISA Servers in an array with NLB enabled on all network interfaces. You have a Network Rule using a Route Relationship between the DMZ network and the External network. The DMZ network is defined as Source Network and the External network is defined as Destination Network in the Network Rule. In the DMZ you have a single server that you access from multiple external clients through an Access Rule in ISA Server. You have observed that only one of the ISA Server computers seems to handle all the load instead of the load being equally distributed between the two ISA Server nodes. If you change the configuration to use a Server Publishing Rule instead of an Access Rule, then you notice that the load is being equally distributed between the two nodes. Based on this you run a “FWEngMon /QueryNLB” from a Command Prompt and find the following (We have only listed the relevant NLB Hook Rules. Please note that the NLB Hook Rules will look different depending on your configuration and Networks definitions in ISA Server):

Hash on source .... (forward): 23.1.1.0--23.1.1.255 -> 0.0.0.0--10.1.0.255

Hash on source .... (forward): 23.1.1.0--23.1.1.255 -> 10.1.2.0--10.255.255.254

Hash on source .... (forward): 23.1.1.0--23.1.1.255 -> 11.0.0.0--23.1.0.255

Hash on source .... (forward): 23.1.1.0--23.1.1.255 -> 23.1.2.0--23.255.255.254

Hash on source .... (forward): 23.1.1.0--23.1.1.255 -> 24.0.0.0--126.255.255.255

Hash on source .... (forward): 23.1.1.0--23.1.1.255 -> 128.0.0.0--255.255.255.254

Hash on destination (reverse): 0.0.0.0--10.1.0.255 -> 23.1.1.0--23.1.1.255

Hash on destination (reverse): 10.1.2.0--10.255.255.254 -> 23.1.1.0--23.1.1.255

Hash on destination (reverse): 11.0.0.0--23.1.0.255 -> 23.1.1.0--23.1.1.255

Hash on destination (reverse): 23.1.2.0--23.255.255.254 -> 23.1.1.0--23.1.1.255

Hash on destination (reverse): 24.0.0.0--126.255.255.255 -> 23.1.1.0--23.1.1.255

Hash on destination (reverse): 128.0.0.0--255.255.255.254 -> 23.1.1.0--23.1.1.255

In this example 23.1.1.0--23.1.1.255 is your DMZ network, and you can therefore see from the above that ISA Server will actually hash the IP affinity based on the IP address of the server in the DMZ and not the IP addresses of the external clients. This means that even if you have thousands of external clients accessing the single server in the DMZ, ISA Server will always hash the affinity based on the DMZ range. If you only have one server in the DMZ that is being accessed, this means that a single IP address is used to make load balancing distribution decisions. In turn explaining why all the traffic is always handled by only one of the ISA Servers.

Because of this you try to change the Access Rule to use a Server Publishing Rule (configured with “Requests appear to come from the original client”) instead, and as mentioned above the load is now distributed equally between the two ISA Server nodes. In order to explain this we run “FWEngMon /QueryNLB” again. We can still see the same rules as in the case when using the Access Rule instead of the Server Publishing Rule. However you will also notice additional NLB Hook Rules that are created specifically for the Server Published Server on IP address 23.1.1.20. As you can now see we hash the IP affinity based on the external IP address ranges instead of the IP address of the published DMZ Server (23.1.1.20) and because of this the load is distributed equally between the two ISA Server nodes.