Bi-Directional Affinity in ISA Server

Bi-Directional Affinity in ISA Server

  • Comments 2
  • Likes

The reason for having a network load balancing (NLB) Bi-Directional Affinity (BDA) feature in ISA Server is that it guarantees that traffic is handled in both directions by the same array server. This is very important and required if you have a Route Relationship (instead of NAT) between two ISA Server networks (for example between the DMZ and the External network) or if you use a Publishing Rule (Server Publishing or Web Publishing) that is configured with “Requests appear to come from the original client” (this is the default setting for a Server Publishing rule) instead of “Requests appear to come from the ISA Server computer” (this is the default setting for a Web Publishing rule). If for example the BDA feature was not available or for some reason not correctly configured in ISA Server, then you could potentially see that incoming traffic from an external client was handled by one array member, while the return traffic from an internal published server (that is configured to route traffic back out through ISA’s internal Virtual NLB IP as it typically will and should be) back to that external client was handled by another array member. This would cause the connection to fail as there is no connection sharing between ISA array members. Because of this we need to guarantee that traffic is handled in both directions by the same array server, and this is what BDA does for us when we have a Route Relationship between two ISA Server networks or if we use “Requests appear to come from the original client” in a Publishing rule.

 

Note that when using a Route Relationship between two ISA Server networks or using “Requests appear to come from the original client” in a Publishing rule, then you should make sure that you enable Integrated NLB on both the Source and Destination network interface for the involved networks. Otherwise BDA will likely fail and you may see strange symptoms. If for example you only enable NLB on one network when having a Route Relationship between two networks, then you should see a “NLB Inconsistent Configuration Detected” ISA Server alert being triggered similar to the following:

 

NLB Inconsistent Configuration Detected

An inconsistency in the Network Load Balancing (NLB) configuration may result in inconsistent handling of traffic between the External network and the DMZ network. When a network rule specifying a route relationship is defined between two networks, NLB must be enabled (or disabled) on both networks. To enable NLB for IPSec remote site networks, enable NLB on the network containing the local tunnel endpoint. To enable NLB for VPN site-to-site and VPN client networks, enable NLB on the selected access networks. Alternatively, for the VPN Client network, you can designate a router for routing traffic according to the static address pool.

 

Using Integrated NLB in ISA Server means that only the NLB single affinity scheme is supported to distribute the load among the array servers. This is a BDA requirement. Single affinity basically means that only a single IP address (source or destination) is used to make load distribution decisions. ISA Server uses NLB Hook Rules to make decisions either it will use the source IP address or destination IP address as affinity, and you can list these NLB Hook Rules by running “FWEngMon /QueryNLB” from a Command Prompt on one of the ISA Servers in the array. That we hash the affinity based on the source IP address for incoming traffic and based on the destination IP address for the return traffic (or vice versa) is actually how BDA works and what guarantees that traffic is handled in both directions by the same array server.

 

FWEngMon.exe is available both for ISA Server 2006 and ISA Server 2004 and can be downloaded here:

 

http://www.microsoft.com/downloads/details.aspx?familyid=01fc5551-5d44-4a99-966a-bd86caeb43d7&displaylang=en

 

http://www.microsoft.com/downloads/details.aspx?familyid=f3306399-d4f9-4989-865e-c61f8293c330&displaylang=en

 

When using Integrated NLB, ISA Server works with Windows NLB to automatically configure Bi-Directional Affinity (BDA). Therefore, you as an ISA Server administrator typically do not have to do any additional steps. You just have to make sure you enable Integrated NLB on the ISA Server network interfaces in ISA Management Console and BDA will work by default and by itself. However there’s always exceptions and you may still experience unexpected behavior as you will see below.

 

Let us take the following scenario to try and explain the above NLB Hook Rules further. Let us say you run two ISA Servers in an array with NLB enabled on all network interfaces. You have a Network Rule using a Route Relationship between the DMZ network and the External network. The DMZ network is defined as Source Network and the External network is defined as Destination Network in the Network Rule. In the DMZ you have a single server that you access from multiple external clients through an Access Rule in ISA Server. You have observed that only one of the ISA Server computers seems to handle all the load instead of the load being equally distributed between the two ISA Server nodes. If you change the configuration to use a Server Publishing Rule instead of an Access Rule, then you notice that the load is being equally distributed between the two nodes. Based on this you run a “FWEngMon /QueryNLB” from a Command Prompt and find the following (We have only listed the relevant NLB Hook Rules. Please note that the NLB Hook Rules will look different depending on your configuration and Networks definitions in ISA Server):

 

Hash on source .... (forward):        23.1.1.0--23.1.1.255      ->         0.0.0.0--10.1.0.255     

Hash on source .... (forward):        23.1.1.0--23.1.1.255      ->        10.1.2.0--10.255.255.254 

Hash on source .... (forward):        23.1.1.0--23.1.1.255      ->        11.0.0.0--23.1.0.255     

Hash on source .... (forward):        23.1.1.0--23.1.1.255      ->        23.1.2.0--23.255.255.254 

Hash on source .... (forward):        23.1.1.0--23.1.1.255      ->        24.0.0.0--126.255.255.255

Hash on source .... (forward):        23.1.1.0--23.1.1.255      ->       128.0.0.0--255.255.255.254

 

Hash on destination (reverse):         0.0.0.0--10.1.0.255      ->        23.1.1.0--23.1.1.255     

Hash on destination (reverse):        10.1.2.0--10.255.255.254  ->        23.1.1.0--23.1.1.255     

Hash on destination (reverse):        11.0.0.0--23.1.0.255      ->        23.1.1.0--23.1.1.255     

Hash on destination (reverse):        23.1.2.0--23.255.255.254  ->        23.1.1.0--23.1.1.255     

Hash on destination (reverse):        24.0.0.0--126.255.255.255 ->        23.1.1.0--23.1.1.255     

Hash on destination (reverse):       128.0.0.0--255.255.255.254 ->        23.1.1.0--23.1.1.255  

 

In this example 23.1.1.0--23.1.1.255 is your DMZ network, and you can therefore see from the above that ISA Server will actually hash the IP affinity based on the IP address of the server in the DMZ and not the IP addresses of the external clients. This means that even if you have thousands of external clients accessing the single server in the DMZ, ISA Server will always hash the affinity based on the DMZ range. If you only have one server in the DMZ that is being accessed, this means that a single IP address is used to make load balancing distribution decisions. In turn explaining why all the traffic is always handled by only one of the ISA Servers.

 

Because of this you try to change the Access Rule to use a Server Publishing Rule (configured with “Requests appear to come from the original client”) instead, and as mentioned above the load is now distributed equally between the two ISA Server nodes. In order to explain this we run “FWEngMon /QueryNLB” again. We can still see the same rules as in the case when using the Access Rule instead of the Server Publishing Rule. However you will also notice additional NLB Hook Rules that are created specifically for the Server Published Server on IP address 23.1.1.20. As you can now see we hash the IP affinity based on the external IP address ranges instead of the IP address of the published DMZ Server (23.1.1.20) and because of this the load is distributed equally between the two ISA Server nodes.

 

Hash on source .... (forward):        23.1.1.0--23.1.1.255      ->         0.0.0.0--10.1.0.255     

Hash on source .... (forward):        23.1.1.0--23.1.1.255      ->        10.1.2.0--10.255.255.254 

Hash on source .... (forward):        23.1.1.0--23.1.1.255      ->        11.0.0.0--23.1.0.255     

Hash on source .... (forward):        23.1.1.0--23.1.1.255      ->        23.1.2.0--23.255.255.254 

Hash on source .... (forward):        23.1.1.0--23.1.1.255      ->        24.0.0.0--126.255.255.255

Hash on source .... (forward):        23.1.1.0--23.1.1.255      ->       128.0.0.0--255.255.255.254

 

Hash on destination (reverse):         0.0.0.0--10.1.0.255      ->        23.1.1.0--23.1.1.255     

Hash on destination (reverse):        10.1.2.0--10.255.255.254  ->        23.1.1.0--23.1.1.255     

Hash on destination (reverse):        11.0.0.0--23.1.0.255      ->        23.1.1.0--23.1.1.255     

Hash on destination (reverse):        23.1.2.0--23.255.255.254  ->        23.1.1.0--23.1.1.255     

Hash on destination (reverse):        24.0.0.0--126.255.255.255 ->        23.1.1.0--23.1.1.255     

Hash on destination (reverse):       128.0.0.0--255.255.255.254 ->        23.1.1.0--23.1.1.255     

 

Hash on source .... (forward):         0.0.0.0--10.1.0.255      ->       23.1.1.20--23.1.1.20      

Hash on source .... (forward):        10.1.2.0--10.255.255.254  ->       23.1.1.20--23.1.1.20      

Hash on source .... (forward):        11.0.0.0--23.1.0.255      ->       23.1.1.20--23.1.1.20      

Hash on source .... (forward):        23.1.2.0--23.255.255.254  ->       23.1.1.20--23.1.1.20      

Hash on source .... (forward):        24.0.0.0--126.255.255.255 ->       23.1.1.20--23.1.1.20      

Hash on source .... (forward):       128.0.0.0--255.255.255.254 ->       23.1.1.20--23.1.1.20     

 

Hash on destination (reverse):       23.1.1.20--23.1.1.20       ->         0.0.0.0--10.1.0.255     

Hash on destination (reverse):       23.1.1.20--23.1.1.20       ->        10.1.2.0--10.255.255.254 

Hash on destination (reverse):       23.1.1.20--23.1.1.20       ->        11.0.0.0--23.1.0.255     

Hash on destination (reverse):       23.1.1.20--23.1.1.20       ->        23.1.2.0--23.255.255.254 

Hash on destination (reverse):       23.1.1.20--23.1.1.20       ->        24.0.0.0--126.255.255.255

Hash on destination (reverse):       23.1.1.20--23.1.1.20       ->       128.0.0.0--255.255.255.254

 

For all other resources in the DMZ network, except the Server Published server, we will however still hash the IP affinity based on the DMZ network range instead of the External IP address ranges. How can we reverse this? We go back to the initial example where we used an Access Rule instead of a Server Publishing rule. Just by modifying the Network Rule so that External is defined as Source Network and DMZ is defined as Destination Network, will we also reverse the NLB Hook Rules. This makes ISA Server hash the IP affinity based on the External IP address ranges instead of the DMZ network range and you should see equal load distribution between the two ISA Server nodes also by using an Access Rule instead of a Server Publishing Rule. Again we run “FWEngMon /QueryNLB” to verify this:

 

Hash on source .... (forward):         0.0.0.0--10.1.0.255      ->        23.1.1.0--23.1.1.255     

Hash on source .... (forward):        10.1.2.0--10.255.255.254  ->        23.1.1.0--23.1.1.255     

Hash on source .... (forward):        11.0.0.0--23.1.0.255      ->        23.1.1.0--23.1.1.255     

Hash on source .... (forward):        23.1.2.0--23.255.255.254  ->        23.1.1.0--23.1.1.255     

Hash on source .... (forward):        24.0.0.0--126.255.255.255 ->        23.1.1.0--23.1.1.255     

Hash on source .... (forward):       128.0.0.0--255.255.255.254 ->        23.1.1.0--23.1.1.255     

 

Hash on destination (reverse):        23.1.1.0--23.1.1.255      ->         0.0.0.0--10.1.0.255     

Hash on destination (reverse):        23.1.1.0--23.1.1.255      ->        10.1.2.0--10.255.255.254 

Hash on destination (reverse):        23.1.1.0--23.1.1.255      ->        11.0.0.0--23.1.0.255     

Hash on destination (reverse):        23.1.1.0--23.1.1.255      ->        23.1.2.0--23.255.255.254 

Hash on destination (reverse):        23.1.1.0--23.1.1.255      ->        24.0.0.0--126.255.255.255

Hash on destination (reverse):        23.1.1.0--23.1.1.255      ->       128.0.0.0--255.255.255.254

 

If you need to reverse the Source and Destination Networks in a Network Rule please note the following: For both NAT relationships and route relationships, traffic can be permitted by configuring Access Rules. Publishing Rules can be configured to allow traffic from the destination network to the source network.

 

Note that in order for ISA Server to be able to create a NLB Hook Rule for a Web Publishing rule using “Requests appear to come from the ISA Server computer”, then name resolution for the name used for the published server under the To tab of the Web Publishing rule must be working. This is of course not required in a Server Publishing Rule since you specify the IP address to Server Publish and not the DNS name of the server. This is probably self explanatory since we need an IP address to be able to create the NLB Hook Rules but we still think it is worth mentioning.

 

Steve Frode Nabseth

Escalation Engineer

Comments
  • I've not seen this occur, but I have wondered about this in production.  With most enterprise-class load-balancers, there are ways to bind the session throughout the infrastructure to a particular server.  To maintain that session affinity, is pretty crucial when you are working in tricky web-app environments.  For example, I work with a bank whose online banking applications require session affinity to assure that one users session doesn't switch over to another users session and suddenly, user a sees user b's bank info.  

    A very large developer of switches and routers have had a similar problem with this recently.  Their large enterprise-class switch had two cpu's that would load-balance applications.  However while each cpu maintained a tcp-port session and usage table, the two cpu's did not.  Therefore, if request Alpha came in and was directed to server Bravo on cpu 1 redirecting traffic to port 5000, and then the reply came back through and hit cpu 2, then another request came in to cpu 2 and was told to hit port 5000, a tcp out of sequence error would occur and both sessions would be reset.  It wasn't a massive outage, as it only occured every now and then.  Session affinity can help resolve this.  That and using ISA for load-balancing versus hardware which can't easily address these issues.

  • SECURITY Security Tip of the Month: Initial Considerations for Secure Deployment http://go.microsoft.com/?linkid=8514049

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment