In this blog post, I would like to talk about a misconfiguration which is still in place on many customer installations. I dealt with many network performance issues where the problem was stemming from using a small MTU size (576 bytes) when communicating with off the subnet hosts.
PMTU discovery option helps communicating endpoints find the most optimum MTU in a TCP session. If this feature is turned off, MTU is set to 576 bytes for all communication with off the subnet hosts. This might badly impact the performance while communicating with the remote hosts.
By default PMTU Discovery is enabled (EnablePMTUDiscovery is set to 1) but due to some older security recommendations, it is set to to 0 as part of server hardening. The reason behind setting that registry key to 0 was to prevent an attacker from forcing Windows to use very small MTU values to decrease the performance.
But that security recommendation is not a valid recommendation anymore as of MS05-019. After that security update, it’s not a security concern anymore because an attacker cannot set MTU size lower than 576 even if PMTU Discovery is enabled. So it shouldn't be set to 0 for security reasons as part of server hardening. This causes performance loss where there's no security concern in terms of small MTU usage.
You can find more information about the changed behavior at the below article:
http://www.microsoft.com/technet/security/bulletin/MS05-019.mspxVulnerabilities in TCP/IP Could Allow Remote Code Execution and Denial of Service (893066)
(From General Information > Vulnerability Details > ICMP Path MTU Vulnerability > Faq for ICMP Path MTU Vulnerability at the above link)
What is Path MTU Discovery?Path maximum transmission unit (PMTU) discovery is the process of discovering the maximum size of packet that can be sent across the network between two hosts without fragmentation (that is, without the packet being broken into multiple frames during transmission). It is described in RFC 1191. For more information, see RFC 1191. For additional information, see the following MSDN Web site.
What is wrong with the Path MTU Discovery process?Path maximum transmission unit (PMTU) discovery allows an attacker to specify a value that can degrade network performance for other connections. On unsecured networks, allowing PMTU discovery carries the risk that an attacker might force the MTU to a very small value and overwork the local system's TCP/IP stack. Normally this behavior would be restricted to the single connection that an attacker could establish. However, this vulnerability allows an attacker to modify the MTU value on other connections beyond their own connection to the affected system.
What does the update do?The update removes the vulnerability by restricting the minimum value of the MTU to 576 bytes. This update also modifies the way that the affected operating systems validate ICMP requests.