ISA Server 2006 SP1 – Problems that Go Beyond the Test Button

ISA Server 2006 SP1 – Problems that Go Beyond the Test Button

  • Comments 2
  • Likes

Introduction

 

ISA Server 2006 SP1 shipped last week and it represents a big milestone for the product. The troubleshooting tools that are now built in will help the administrators effectively test their publishing rules prior to deploy published server to the public. But, what if everything looks good in the “testing environment” and when deploys to the external public it is failing?

 

An ISA administrator was saying that when he used the ISA Server 2006 SP1 Test Button, it shows that the publishing rule was configured correctly; much like Figure 1 below.

 

Figure 1 – Web Publishing Test results.

 

The problem is that when a user connected, he was receiving error (operation timed out) while accessing the web site from the Internet.

 

What Should I Expect from Test Button?

 

The first thing we need to do is set proper expectations about the Test Button feature.  Here are some key aspects of this option:

 

—  What does it do?

1.       Validates that TCP/IP connection can be established

-          Hostname is resolvable

-          Host is accessible on ports defined in the rule Bridging tab

 

 

2.       Verifies server certificate information

-          Cert not expired, not revoked

-          Cert issued by a trusted CA

-          Cert issued to the right name (with SAN support)

 

 

3.       Validates that an HTTP GET request can pass through:

-          ISA auth-delegation compatible with server auth requirements

-          Published paths exist (tested only of auth not required)

-          Published server accepts names forwarded in host header (as configured in the rule To tab)

 

—  What are the known limitations?

1.       Internal rule conflicts are not checked

-          It doesn’t test authentication; it only tests that IIS / ISA authentication methods are compatible

 

 

Web Publishing Scenario

 

The scenario was a basic HTTP web publishing rule with no SSL involved. The Web Server was using IIS7 and was working just fine for internal users.

 

 

Figure 2 – Web Publishing Scenario

 

Note: the dotted line represents the uplink between the switches.

 

Gathering Data

 

The first troubleshooting step we performed was to understand why the client was getting time out from outside. To do that we enabled Live Logging and found the following error:

 

 

Figure 3 – Live logging error while accessing the IIS Server.

 

The 10060 is clear by definition: ISA couldn’t get an answer from the published server. But this was really trick since we could access the web server from the ISA Server itself. To really understand what was happening it was necessary to attach netmon to the internal nic of the ISA Server.

 

Note: you can also use ISA Data Packager in repro mode that already enables netmon and gathers additional information that can be used for troubleshooting purpose.

 

Analyzing the Netmon Result

 

After starting the netmon trace and reproducing the issue from the outside client, the following behavior was noticed:

 

1. ISA Server sends the TCP SYN to establish the handshake on port 80:

11:50:35.510      192.168.0.107     10.20.20.138      TCP   TCP: Flags=.S......, SrcPort=1192, DstPort=HTTP(80), Len=0, Seq=3373035932, Ack=0, Win=65535 (scale factor not found)

 

2. ISA Server does no receive an answer and try it again:

11:50:38.474      192.168.0.107     10.20.20.138      TCP   TCP: Flags=.S......, SrcPort=1192, DstPort=HTTP(80), Len=0, Seq=3373035932, Ack=0, Win=65535 (scale factor not found)

 

3. …and again:

11:50:44.483      192.168.0.107     10.20.20.138      TCP   TCP: Flags=.S......, SrcPort=1192, DstPort=HTTP(80), Len=0, Seq=3373035932, Ack=0, Win=65535 (scale factor not found)

 

4….and again:

11:50:56.520      192.168.0.107     10.20.20.138      TCP   TCP: Flags=.S......, SrcPort=1192, DstPort=HTTP(80), Len=0, Seq=266374158, Ack=0, Win=65535 (scale factor not found)

 

5. After all those attempts it times out:

11:51:26.493      192.168.0.107     10.20.20.138      TCP   TCP: Flags=.S......, SrcPort=1192, DstPort=HTTP(80), Len=0, Seq=327351070, Ack=0, Win=65535 (scale factor not found)

 

It seems clear from the network capture that the IIS Server is not answering the request.  Oddly enough, the IIS admin responded that the IIs server was answering the request, and provided this capture from the IIS server:

 

1. IIS Server receives the TCP SYN on port 80:

192.168.0.107     10.20.20.138      TCP   TCP: Flags=.S......, SrcPort=1198, DstPort=HTTP(80), Len=0, Seq=333322946, Ack=0, Win=65535 (scale factor not found)

 

2. It answers the SYN with the ACK:

10.20.20.138      192.168.0.107     TCP   TCP: Flags=.S..A..., SrcPort=HTTP(80), DstPort=1198, Len=0, Seq=1649575802, Ack=333322947, Win=8192 (scale factor not found)

 

3. It receives another SYN:

192.168.0.107     10.20.20.138      TCP   TCP: Flags=.S......, SrcPort=1198, DstPort=HTTP(80), Len=0, Seq=333322946, Ack=0, Win=65535 (scale factor not found)

 

4. It answers again”

29    10:00:07.573      10.20.20.138      192.168.0.107     TCP   TCP: Flags=.S..A..., SrcPort=HTTP(80), DstPort=1198, Len=0, Seq=1649575802, Ack=333322947, Win=8192 (scale factor not found)

 

5. ..and this happens until IIS resets the connection since the handshake was not established successfully:

 

192.168.0.107     10.20.20.138      TCP   TCP: Flags=.S......, SrcPort=1198, DstPort=HTTP(80), Len=0, Seq=333322946, Ack=0, Win=65535 (scale factor not found)

 

10.20.20.138      192.168.0.107     TCP   TCP: Flags=.S..A..., SrcPort=HTTP(80), DstPort=1198, Len=0, Seq=1649575802, Ack=333322947, Win=8192 (scale factor not found)

 

192.168.0.107     10.20.20.138      TCP   TCP: Flags=.S......, SrcPort=1198, DstPort=HTTP(80), Len=0, Seq=3699107062, Ack=0, Win=65535 (scale factor not found)

 

10.20.20.138      192.168.0.107     TCP   TCP: Flags=.S..A..., SrcPort=HTTP(80), DstPort=1198, Len=0, Seq=1649575802, Ack=333322947, Win=8192 (scale factor not found)

 

10.20.20.138      192.168.0.107     TCP   TCP: Flags=..R....., SrcPort=HTTP(80), DstPort=1198, Len=0, Seq=1649575803, Ack=333322947, Win=0 (scale factor not found)

 

The question that arises now is: if ISA Server is sending packets to the IIS server and the IIS server is replying, then where is the traffic failing between ISA and IIS? Before we think too deeply on that question, we’ll need to step back and understand the topology. The external client is identified in this lab as 192.168.0.107 and the reason why the IIS Server is receiving this address as the source IP in the packet is because of the following option that it is selected in the publishing rule:

 

Figure 4 – ISA Server Publishing option.

 

According to the customer, this option was selected because they want to audit IIS Logs and know what external IPs is accessing the web site.  A couple of tests were performed at the IIS server to better understand how the IIS sends traffic to non-local subnets:

 

C:\Users\administrator.CONTOSO>tracert -d 192.168.0.107

 

Tracing route to 192.168.0.107 over a maximum of 30 hops

 

  1     *        *        *     Request timed out.

 

It timed out from  the beginning, so it was time to check the routing table:

 

C:\Users\administrator.CONTOSO>route print

===========================================================================

Interface List

 10 ...00 03 ff 06 fc 23 ...... Intel 21140-Based PCI Fast Ethernet Adapter (E

lated)

  1 ........................... Software Loopback Interface 1

 11 ...00 00 00 00 00 00 00 e0  isatap.{7B34369C-34DA-4775-9C7B-1CFEAF83257A}

 12 ...02 00 54 55 4e 01 ...... Teredo Tunneling Pseudo-Interface

===========================================================================

 

IPv4 Route Table

===========================================================================

Active Routes:

Network Destination        Netmask          Gateway       Interface  Metric

          0.0.0.0          0.0.0.0      10.20.20.5      10.20.20.138    276

       10.20.20.0    255.255.255.0         On-link      10.20.20.138    276

     10.20.20.138  255.255.255.255         On-link      10.20.20.138    276

     10.20.20.255  255.255.255.255         On-link      10.20.20.138    276

        127.0.0.0        255.0.0.0         On-link         127.0.0.1    306

        127.0.0.1  255.255.255.255         On-link         127.0.0.1    306

  127.255.255.255  255.255.255.255         On-link         127.0.0.1    306

        224.0.0.0        240.0.0.0         On-link         127.0.0.1    306

        224.0.0.0        240.0.0.0         On-link      10.20.20.138    276

  255.255.255.255  255.255.255.255         On-link         127.0.0.1    306

  255.255.255.255  255.255.255.255         On-link      10.20.20.138    276

===========================================================================

Persistent Routes:

  Network Address          Netmask  Gateway Address  Metric

          0.0.0.0          0.0.0.0      10.20.20.5  Default

===========================================================================

 

The answer for this problem it is showed above in red, the default gateway. Now for sure we know what was happening. Check the traffic flow in the diagram below:

 

 

Figure 5 – Traffic flow.

 

As you can see above, client sends the HTTP GET, the ISA Server accepts the request since it matches with the web publishing rule, it tries to establish the TCP handshake on port 80 with the IIS Server sending the TCP SYN, the IIS Server look to the source IP and notice that it is an external IP and sends the TCP SYN ACK to the default gateway which is the router.

 

 

Conclusion

 

The conclusion here is that the switch layer 3 that the IIS Server was using as default gateway didn’t have the correct routing table configured to allow the external IP go back through the ISA Server. The possible solutions were:

—  Change the IIS Server to use ISA Server as default gateway.

—  Add specific router in the layer 3 switch for the internal networks and change the layer 3 switch to use ISA Server as default gateway.

—  Change the ISA Server publishing rule to allow the requests appears that are coming from the ISA Server.

 

The first option was the one choose by the customer and it fixed the issue. The lesson learned here are:

—  The Test Button is a great resource to validate the configuration. But since the traffic is NOT 100% the same as when the client access from outside, the test can’t check everything.

—  A good understanding of your own network topology could avoid issues like that to happen.

 

 

Author

Yuri Diogenes

Security Support Engineer – Microsoft CSS Forefront (ISA/TMG) Team

 

Technical Reviewers

Jim Harrison

Microsoft Forefront (ISA/TMG) Sustained Engineering Team

 

Doron Juster

Microsoft Forefront (ISA/TMG) Sustained Engineering Team

 

 

Comments
  • Thanks a lot, Yuri! Great article with detailed explanation of the Test Button functions.

  • Thanks a lot, Yuri! Great article with detailed explanation of the test button functioning.

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment