In recent months, we have seen several instances of tunneled applications failures, which seems to affect a limited number of an organization’s clients. In all cases, these are IAG Servers that have been working well, but suddenly, some customers are unable to use the tunneled apps (applications that are client/server based, and require the use of the SSL VPN component to function). Investigation of this issue has revealed an interesting fact. As it turns out, some Internet Service Providers (ISPs) have started employing a technique known as “NXDomain Manipulation”, which leads to this, as well as other problems. This is still not done by all of them, but some countries are more affected by others.

NXDomain manipulation is a process in which the ISP changes the normal behavior of DNS servers. One of the primary principles of the DNS system is the RCODE, or Response-code. As specified in section 4.1.1 of RFC 1035 (http://tools.ietf.org/html/rfc1035), DNS servers respond with an RCODE of 3 (“Name Error”, a.k.a. NXDOMAIN) when a client submits a query for a name that does not exist. Some ISPs are configuring their DNS to respond differently, so when the servers receive a request for a non-existent domain, instead of replying with an RCODE of 3, they respond with an RCODE of 0 (“No error”) and direct the browser to an internal webpage hosted by the ISP. That page would typically be either a search page or a custom error page such as the one shown below:

clip_image002[5]

Typical error page when NXDomain manipulation is employed

ICANN is not happy about this practice, and published a draft-memo about it (http://www.icann.org/en/topics/new-gtlds/nxdomain-substitution-harms-24nov09-en.pdf). In the memo, the organization attempts to ban NXDomain manipulation, saying “ICANN strongly discourages the use of DNS redirection, wildcards, synthesized responses and any other form of NXDOMAIN substitution in new and existing gTLDs and ccTLDs and any other level in the DNS tree for registry-class domain names”. ICANN’s plan is to include this in the agreement that owners of the new gTLDs would have to sign.

For regular users, NXDomain manipulation is not a problem, and might even be helpful, as the result is easier to digest than a bleak numerical error message, but for IAG customers, this can lead to some unpleasantness. The root cause of the issue is the way the IAG Tunneling components perform name resolution as part of their design. This design is based on that when a tunneled application tries to contact its server, the main DNS server would not be able to resolve that servers name, but when NXDomain manipulation is involved, DNS is able to resolve it…or at least, it looks that way. In reality, though, the communication process is interrupted by this false-positive, and the application tries to contact the IP resolved by the ISPs DNS servers, which fails, of course.

Unfortunately, the way this occurs, from the IAG’s perspective, is exactly as it should, and so there’s no way to fix it. An ISP that performs NXDomain manipulation will prevent its users from using tunneled applications, and the user can only work around it, and not solve it (assuming, of course, that they are not willing to switch to another ISP).

clip_image004[4]

Typical error 3 response by a DNS server to a non-existent page.

clip_image006[4]

Success replied by manipulated DNS server. Note that the server gives an IP array of its internal search pages.

The available workarounds are the following:

1. Some ISPs allow their customers to ask to be excluded from the process of NXDomain manipulation. The exclusion is sometimes based on a cookie implanted on the user’s computer, and in other cases, based on the computers MAC address. Technically, a cookie-based exclusion would not solve the problem, as the tunneled app that performs the DNS query would typically not be able to send the cookies to the DNS server to get excluded.

2. Since this manipulation is done by the ISPs server, using a different DNS server that does not do NXDomain manipulation is also a workaround. Verizon hosts publicly usable DNS Server at the IP address 4.2.2.1 and 4.2.2.2, and these can be used in this scenario. To use these servers, the user needs to change his networking configuration to these DNS servers, and that solves the problem well. However, when the affected computer is a mobile computer, this can complicate things, because if the computer is manually set to a specific DNS (as opposed to a DHCP-assigned one), it won’t be able to resolve names when the user connects to his workplace’s corporate network, which has its own DNS servers. A solution we have come up with for this is a small application that allows the user to change the DNS settings on his computer from automatic to static with one mouse-click (contact Microsoft Support to obtain this tool).

clip_image008[4]

DNS Switcher app

It’s also possible to create a custom VBScript that changes these settings, and implement it as a custom application on the IAG portal, though we do not have a prepared guide for this.

3. A variation of the above workaround is to set the router at the user’s residence to assign connecting users a static DNS server address (like the 4.2.2.1/4.2.2.2 we just mentioned), instead of the one that is assigned by the ISP. This, naturally, may be challenging, as routers have very diverse and often unique user-interface. This could be too complicated for home users to perform by themselves.

clip_image010[4]

Typical DNS settings on a home router

4. Another workaround is to connect to the corporate network using Network Connector instead of a tunneled application. When using the Network Connector, the user’s computer has a full VPN access to the network, including a change to the DNS settings, so it bypasses the NXDomain manipulation completely. Not all organizations are OK with using the Network Connector, because it opens connectivity to internal network that may not meet the organizations security policy, but technically, this is a good solution.

What else can one do? Well, hopefully, ISPs will start stepping back from this technique, following the action taken by ICANN. You might be able to help by applying pressure to your ISP to stop this as well, by showing them that this practice may be more upsetting than helpful.