Consider a scenario where a TMG administrator has configured their TMG Server 2010 installed on a Windows Server 2008 R2 for inbound VPN connections. External VPN users (two or more users) are behind a NAT device, which NATs all outbound L2TP VPN traffic. When users try to connect using L2TP VPN connections, only one user from this network can connect at a time. Every connection attempt from another user fails with the following error:
Error message: error 809
The network connection between your computer and the VPN server could not be established because the remote server is not responding. This could be because one of the network devices (e.g., firewalls, NAT, Router etc.) between your computer and the remote server is not configured to allow VPN connections. Please contact your Administrator or your service provider to determine which device may be causing the problem.
In this case, since users can connect to VPN as long as they are unique from behind a NAT device, the basic VPN configuration on the TMG Server would not have any problem.
To verify if the issue is specific to L2TP VPN, we configured the TMG VPN Server to allow PPTP. Tests from the client end revealed that we could successfully establish more than one PPTP VPN connections. So the test confirmed that the issue was specific to L2TP VPN connections.
Since in such a scenario, traffic would appear to come from the same external IP, it is normal to think that the TMG server might be dropping the connections as per its Flood Mitigation settings. We did notice some TCP/IP Connection Limit Exceeded errors on the TMG Server in this case. So we created an exception list for the incoming IP address in the Flood Mitigation Settings. Though we see the TCP/IP connection Limit errors go away, this did not resolve our issue.
TMG Live logging shows Initiated and Closed Connections without much detail. Network traces do not help much in this case as the traffic is L2TP and encrypted.
To isolate whether the issue was related to TMG or RRAS, we set up a parallel VPN setup with a Windows Server 2008 R2 machine, configured as RRAS. We were able to reproduce the issue successfully without installing TMG on this machine.
The problem in our scenario turned out to be specific to L2TP VPN traffic from behind a NAT device to Windows Server 2008 R2 RRAS Server. This was identified as a problem in Windows Server 2008 R2 and the following KB article addresses this issue. The public hotfix included within the article below updates the Fwpkclnt.sys and Tcpip.sys files:
Only one of the clients that are behind the same NAT device can create L2TP VPN connections to a VPN server that is running Windows Server 2008 R2
Author Niladri Dasgupta Support Engineer Microsoft CSS Forefront TMG Team
Technical Reviewers Yuri Diogenes Sr. Support Escalation Engineer Microsoft CSS Forefront TMG Team
Mohit Kumar Sr. Support Escalation Engineer Microsoft CSS Forefront TMG Team
I do not understand the resolution. I am experiencing a similar problem in that we are using the TMG for VPN access. The TMG 2010 is installed on a VM of Server 2008 R2 using Hyper-V. Is it the Server 2008 R2 that is blocking the connections or the TMG? If so, how can I fix it?