Common Networking Issues After Applying Windows Server 2003 SP2 on SBS

[Today's post comes to us courtesy of Mark Stanfill]

Overview

We have seen an increasing number of support calls from customers experiencing a variety of networking-related issues after installing Windows Server 2003 SP2.  We have previously covered this topic in this blog (here), and we wanted to come back to the topic now that we have a better idea of the scope of the issue, what causes it, and a better idea of how to fix these issues.  The issues discussed here are not unique to SBS, but they do tend to be more common in SBS networks because of the large number of SBS 2003 servers used for NAT (either RRAS or ISA).  There are a smaller number of issues that can appear even in single-NIC scenarios.  In most of the cases we've seen, updating the NIC drivers fixes the issue, but there are a significant portion of these cases where RSS and TCP Chimney Offload (TaskOffload) need to be disabled via the registry as well.

Background

Windows Server 2003 SP2 introduces a number of new networking features, including TCP Chimney Offload, Receive Side Scaling (RSS), and Network Direct Memory Access (NetDMA).  Unfortunately, RSS and TCP Chimney Offload are not compatible with these technologies:

• Windows Firewall

• Internet Protocol security (IPsec)

• Internet Protocol Network Address Translation (IPNAT)

• Third-party firewalls

• NDIS 5.1 intermediate drivers

We are researching a solution and expect to have an update available soon.

Symptoms and Issues We've Seen

I have to be careful here - this is a list of symptoms that we've seen where the troubleshooting tips below resolved the issue, but there are a number of caveats here.  First, not all of these have been verified.  For each of these symptoms, there are a huge number of other potential causes.  The steps in the troubleshooting section below should only be used in cases where you are seeing one of the symptoms below and the only thing that has changed is that you've recently installed SP2.

• Unable to VPN to the Server ("Error 800: Unable to establish connection").
• Unable to RDP to SBS server
• Unable to connect to shares on SBS server from the LAN
• Unable to join a client machine to the domain
• Unable to connect to Exchange from Outlook
• Unable to connect to SSL sites either on the SBS server or on the Internet (including CompanyWeb)
• Slow network performance
• Outgoing FTP connections fail
• DHCP Server service crashes
• Slow domain logins
• Intermittent connection failures from NAT clients behind the server
• Intermittent RPC communications failures

Troubleshooting

The critical question in determining which steps to use is "how many NICs are in the box?".  If you have a multi-homed box (more than 1 NIC), use all four steps below.  If you have a single NIC SBS server, our recommendation is to follow the steps below in order and see if each step provides resolution.  RSS can provide significant performance enhancements if your network hardware supports it end-to-end.  Our general recommendation is to update your NIC driver in 100% of cases. 

Step 1: Update the Driver

Most of the issues we've seen are related to older NIC drivers that do not know how to use the advanced networking features ( of Windows Server 2003 SP2.  Virtually every major manufacturer has come out with a new driver in '07.  Before you do anything else, make sure that you have the latest Server 2003 drivers for you NIC.  The vast majority of the cases we've seen can be solved by this step alone. 

Step 2: Disable Offloading on the Advanced Properties of the NIC

Most NICs have various offloading functions that can increase network performance (or at least lower CPU usage on the server).  Again, this is only if your network hardware supports high throughput end-to-end.  That means that your NICs, cabling, switches, and possibly routers all have to support gigabit networking and know how to deal with these offloading functions.  Your vendor(s) have the final say on wether RSS, checksum offloading, etc. will work with the combination of equipment you have.  That's a nice way of saying that your typical "lean and mean" small business is not likely to have invested in high-end hardware.  In support, we routinely turn these functions off when troubleshooting any networking issues.  We've never seen a case in an SBS environment where there was perceived network slowness after disabling offload functions.  If in doubt, disable all advanced features, test to see if they provide relief, and then re-enable them one by one to see if A) these features cause your networking issue and B) if they offer any kind of performance gain.  Here are a few examples (your vendor will likely have different settings based on model and driver revision)

Step 3: Disable RSS in the Registry

Use the steps in KB 927695 to disable Receive Side Scaling (RSS) by adding a DWORD registry key value for

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableRSS and setting it to 0.  A reboot is required to make the value go in to effect.

(Like the KB article says, usual caveats about having a backup, etc apply before making any registry changes).

Step 4: Set DisableTaskOffload in the Registry

Use the steps in KB 904946 to create a DWORD value for

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\DisableTaskOffload and set it to 1.  A reboot is required to make this value go in to effect.