Hi,
In today's blog, I'll talk about an MTU issue that occurs on Windows Vista onwards (Vista/7/2008/2008 R2).
One of our customers reported that their SMTP server (running on Windows 2008) was failing to send e-mails to certain remote SMTP servers because e-mail delivery was disrupted at transport layer.
After analyzing the network trace collected on the source Windows 2008 Server, we found out that the remote system was offering a TCP MSS size of 512 bytes and Windows 2008 server kept sending the data packets with an MSS size of 536 bytes. As a result, those packets weren't succesfully delivered to the remote system. You can find more details about the problem and root cause below:
Note: IP addresses and mail server names are deliberately changed.
Source SMTP server: 10.1.1.1 - mailgateway.contoso.comTarget SMTP server: 10.1.1.5 - mailgateway2.contoso.com
a) Source SMTP server establishes TCP 3-way handshake with the target SMTP server. Source server suggests an MSS size of 1460 bytes and the target suggests an MSS size of 512 bytes:
No. Time Source Destination Protocol Info 1 0.000000 10.1.1.1 10.1.1.5 TCP 28474 > 25 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=8 2 0.022001 10.1.1.5 10.1.1.1 TCP 25 > 28474 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=512 3 0.000000 10.1.1.1 10.1.1.5 TCP 28474 > 25 [ACK] Seq=1 Ack=1 Win=65392 Len=0
b) Then data starts flowing. Under normal circumstances, the minimum of MSS will be selected as the MSS of the given TCP session by both parties and it will be used throughout the session.
No. Time Source Destination Protocol Info 4 0.075005 10.1.1.5 10.1.1.1 SMTP S: 220 mailgateway2.contoso.com ESMTP Tue, 20 Apr 2010 15:18:42 +0200 5 0.001000 10.1.1.1 10.1.1.5 SMTP C: EHLO Mailgateway.contoso.com 6 0.021001 10.1.1.5 10.1.1.1 SMTP S: 250-mailgateway2.contoso.com Hello Mailgateway.contoso.com [10.1.1.1] | 250-SIZE 26214400 | 250-PIPELINING | 250 HELP 7 0.001000 10.1.1.1 10.1.1.5 SMTP C: MAIL FROM:<postmaster@contoso.com> SIZE=2616 | RCPT TO:<test@test.abc.com> 8 0.183011 10.1.1.5 10.1.1.1 SMTP S: 250 OK | 250 Accepted 9 0.000000 10.1.1.1 10.1.1.5 SMTP C: DATA 10 0.022001 10.1.1.5 10.1.1.1 SMTP S: 354 Enter message, ending with "." on a line by itself
c) Even though an MSS size of 512 should be commonly agreed by both parties, Windows 2008 server doesn't seem to be using that value and keeps sending data with an MSS of 536 bytes:
No. Time Source Destination Protocol Info 11 0.294017 10.1.1.1 10.1.1.5 SMTP C: Message Body, 536 bytes
d) Most likely the TCP segment with 536 bytes of data doesn't arrive at the target server and we don't get a TCP ACK back as a result so we start TCP packet retransmissions:
No. Time Source Destination Protocol Info 12 0.600034 10.1.1.1 10.1.1.5 SMTP [TCP Retransmission] C: Message Body, 536 bytes 13 0.190011 10.1.1.5 10.1.1.1 SMTP [TCP Retransmission] S: 354 Enter message, ending with "." on a line by itself 14 0.000000 10.1.1.1 10.1.1.5 TCP [TCP Dup ACK 12#1] 28474 > 25 [ACK] Seq=649 Ack=269 Win=65124 Len=0 15 1.010058 10.1.1.1 10.1.1.5 SMTP [TCP Retransmission] C: Message Body, 536 bytes 16 2.400137 10.1.1.1 10.1.1.5 SMTP [TCP Retransmission] C: Message Body, 536 bytes 17 4.800274 10.1.1.1 10.1.1.5 SMTP [TCP Retransmission] C: Message Body, 536 bytes
e) Finally the source server closes the TCP session as it fails to successfully deliver the 536 bytes TCP segment to the target system:
No. Time Source Destination Protocol Info 18 9.600550 10.1.1.1 10.1.1.5 TCP 28474 > 25 [RST, ACK] Seq=649 Ack=269 Win=0 Len=0
The same problem doesn't happen if the source server is a Windows 2003 server.
After explaining the problem, now let's try to understand the root cause:
This issue stems from the fact that Windows Vista onwards systems don't accept an MTU size lower than 576 bytes:
TCP/IP Registry Values for Microsoft Windows Vista and Windows Server 2008http://www.microsoft.com/downloads/details.aspx?familyid=12AC9780-17B5-480C-AEF7-5C0BDE9060B0&displaylang=en
MTUKey: Tcpip\Parameters\Interfaces\interfaceGUID Value Type: REG_DWORD—numberValid Range: From 576 to the MTU of the underlying networkDefault: 0xFFFFFFFFDescription: This value overrides the default Maximum Transmission Unit (MTU) for a network interface. The MTU is the maximum IP packet size, in bytes, that can be transmitted over the underlying network. For values larger than the default for the underlying network, the network default MTU is used. For values smaller than 576, the MTU of 576 is used. This setting only applies to IPv4.Note: Windows Vista TCP/IP uses path MTU (PMTU) detection by default and queries the network adapter driver to find out what local MTU is supported. Altering the MTU value is typically not necessary and might result in reduced performance.
Since minimum MTU that could be used by a Window Vista onwards system is 576 bytes, a TCP MSS (maximum segment size) should be 536 bytes at miminum so that's why Windows 2008 source server tries to send TCP segments with 536 bytes of data. TCP MSS value is calculated as follows:
TCP MSS = IP MTU - IP header size (20 bytes by default) - TCP header size (20 bytes by default)
Hope this helps
Thanks,Murat
Hi there,
In this blog post, I would like to talk about a named pipe access issue on Windows 2008 that I had to deal with recently. One of our customers was having problems in accessing named pipes anonymously on Windows 2008 and therefore we were involved in to address the issue. Even the required configuration for anonymous pipe access was in place, the pipe client was getting ACCESS DENIED when trying to access the pipe.
The problem was easy to reproduce on any Windows Vista or later system. Just run a named pipe server application which creates a pipe, then try to connect to the pipe anonymously from a remote system. You can see more details below on how to reproduce this behavior:
a) Compile the sample pipe server&client application given at the following MSDN link:
http://msdn.microsoft.com/en-us/library/aa365588(VS.85).aspx Multithreaded Pipe Serverhttp://msdn.microsoft.com/en-us/library/aa365592(VS.85).aspx Named Pipe Client
b) Add the named pipe created by Pipe server to the Null session pipe lists (configuring "Nullsessionpipes" registry key under LanmanServer)
c) Do not enable “Network access: Let Everyone permissions apply to anonymous users” from local GPO or domain GPO
d) Make sure that the Pipe ACL included Anonymous user with Full Control permission. (You can do that by using a 3rd party application like pipeacl.exe)
e) Start a command line within the Local System account security context by running a command similar to below:
at 12:40 /interactive cmd.exe
f) Run the pipe client application from the command line and try to connect to the pipe. You'll get an ACCESS_DENIED in response from the server.
Note that as soon as you re-enable “Network access: Let Everyone permissions apply to anonymous users”, pipe client starts successfully opening the pipe and reading from/writing to pipe.
UNDERSTANDING THE ROOT CAUSE:==============================
Well now after explaining the problem, now let's take a look at the root cause of this problem:
Note: Some of the outputs below are WinDBG (debugger) outputs.
1) From Vista onwards, in order to access an object, you need to pass two security checks:
a) Integrity check => For Vista onwards
b) Classical access check (checking object’s security descriptor against the desired access) => For all Windows versions
Note: Integrity check couldn’t be turned off even if you disable UAC (and we wouldn’t want to do that either)
2) In test pipe application, we see the following differences when “Network access: Let Everyone permissions apply to anonymous users” is enabled and disabled:
a) “Network access: Let Everyone permissions apply to anonymous users” ENABLED situation
=> Token of the thread:
- The user is anonymous (as expected)
- The token has also the SID S-1-16-8192 (which represents Medium integrity level). So the thread will be accessing the pipe object while its integrity level is Medium
kd> !token -n
_ETHREAD 892ef810, _TOKEN 9a983b00
TS Session ID: 0
User: S-1-5-7 (Well Known Group: NT AUTHORITY\ANONYMOUS LOGON)
Groups:
00 S-1-0-0 (Well Known Group: localhost\NULL SID)
Attributes -
01 S-1-1-0 (Well Known Group: localhost\Everyone)
Attributes - Mandatory Default Enabled
02 S-1-5-2 (Well Known Group: NT AUTHORITY\NETWORK)
03 S-1-5-15 (Well Known Group: NT AUTHORITY\This Organization)
04 S-1-5-64-10 (Well Known Group: NT AUTHORITY\NTLM Authentication)
05 S-1-16-8192 Unrecognized SID
Attributes - GroupIntegrity GroupIntegrityEnabled
Primary Group: S-1-0-0 (Well Known Group: localhost\NULL SID)
Privs:
23 0x000000017 SeChangeNotifyPrivilege Attributes - Enabled Default
Authentication ID: (0,15a3fe2)
Impersonation Level: Impersonation
TokenType: Impersonation
Source: NtLmSsp TokenFlags: 0x2000
Token ID: 15a3fe5 ParentToken ID: 0
Modified ID: (0, 15a3fe8)
RestrictedSidCount: 0 RestrictedSids: 00000000
OriginatingLogonSession: 0
=> Security descriptor of the pipe (DACL & SACL of the pipe object)
- Anonymous Logon has full access to the pipe object
- Integrity level’s of objects are stored in the SACL of the security descriptor of the object. If the integrity level is not explicitly assigned, the object’s integrity level is Medium.
kd> !sd 0x81f69848 1
->Revision: 0x1
->Sbz1 : 0x0
->Control : 0x8004
SE_DACL_PRESENT
SE_SELF_RELATIVE
->Owner : S-1-5-32-544 (Alias: BUILTIN\Administrators)
->Group : S-1-5-21-1181840707-4124064209-3703316816-513 (no name mapped)
->Dacl :
->Dacl : ->AclRevision: 0x2
->Dacl : ->Sbz1 : 0x0
->Dacl : ->AclSize : 0x5c
->Dacl : ->AceCount : 0x4
->Dacl : ->Sbz2 : 0x0
->Dacl : ->Ace[0]: ->AceType: ACCESS_ALLOWED_ACE_TYPE
->Dacl : ->Ace[0]: ->AceFlags: 0x0
->Dacl : ->Ace[0]: ->AceSize: 0x18
->Dacl : ->Ace[0]: ->Mask : 0x001f01ff
->Dacl : ->Ace[0]: ->SID: S-1-5-32-544 (Alias: BUILTIN\Administrators)
->Dacl : ->Ace[1]: ->AceType: ACCESS_ALLOWED_ACE_TYPE
->Dacl : ->Ace[1]: ->AceFlags: 0x0
->Dacl : ->Ace[1]: ->AceSize: 0x14
->Dacl : ->Ace[1]: ->Mask : 0x001f01ff
->Dacl : ->Ace[1]: ->SID: S-1-5-7 (Well Known Group: NT AUTHORITY\ANONYMOUS LOGON)
->Dacl : ->Ace[2]: ->AceType: ACCESS_ALLOWED_ACE_TYPE
->Dacl : ->Ace[2]: ->AceFlags: 0x0
->Dacl : ->Ace[2]: ->AceSize: 0x14
->Dacl : ->Ace[2]: ->Mask : 0x00120089
->Dacl : ->Ace[2]: ->SID: S-1-1-0 (Well Known Group: localhost\Everyone)
->Dacl : ->Ace[3]: ->AceType: ACCESS_ALLOWED_ACE_TYPE
->Dacl : ->Ace[3]: ->AceFlags: 0x0
->Dacl : ->Ace[3]: ->AceSize: 0x14
->Dacl : ->Ace[3]: ->Mask : 0x001f01ff
->Dacl : ->Ace[3]: ->SID: S-1-5-18 (Well Known Group: NT AUTHORITY\SYSTEM)
->Sacl : is NULL
So in this scenario, a thread with integrity level of Medium is accessing an object with an integrity level of Medium. Hence it’s ok from Integrity check perspective to access the object. Once the integrity check is passed, DACL evaluation is made next (the classical access check that is done in all Windows versions). Since Anonymous user has access on the DACL of the pipe, it passes that stage as well and access to the pipe object is granted.
b) “Network access: Let Everyone permissions apply to anonymous users” DISABLED situation
- The token has also the SID S-1-16-0 (which represents Untrusted integrity level). So the thread will be accessing the pipe object while its integrity level is Untrusted
_ETHREAD 892dab58, _TOKEN 9a81b7f8
01 S-1-5-2 (Well Known Group: NT AUTHORITY\NETWORK)
02 S-1-5-15 (Well Known Group: NT AUTHORITY\This Organization)
03 S-1-5-64-10 (Well Known Group: NT AUTHORITY\NTLM Authentication)
04 S-1-16-0 Unrecognized SID
Authentication ID: (0,15a3909)
Source: NtLmSsp TokenFlags: 0x0
Token ID: 15a390c ParentToken ID: 0
Modified ID: (0, 15a390f)
- Integrity level’s of objects are stored in the SACL of the security descriptor objects. If the integrity level is not explicitly assigned, the object’s integrity level is Medium.
So in this scenario, a thread with integrity level of Untrusted is accessing an object with an integrity level of Medium. Hence it’s NOT OK from Integrity check perspective to access the object and integrity check mechanism denies access to the object. Classical DACL evaluation is even not done here.
In summary,
anonymous pipe access fails because of integrity check. When “Network access: Let Everyone permissions apply to anonymous users” is enabled, the EVERYONE SID is also added to the thread token and hence the token’s integrity level is raised (to medium in this scenario). So integrity check succeeds when this policy is enabled.
HOW TO FIX IT:
============
1) The most meaningful solution here is to set the Pipe object’s integrity level to Untrusted. If we could achieve this, we should be able to pass the integrity check because the integrity level of both the token and the object that the token was trying to open (with Read/Write permissions) would be the same (untrusted)
2) Changing the pipe object’s integrity level could be achieved in two different ways:
a) Setting the integrity level while creating the PIPE from the server application (via CreateFile() API)
b) Setting the integrity level after the PIPE is created (via SetSecurityInfo() API) (Either from the server application or from another application)
3) While searching for possible programmatic solutions, we have come across a very good source code example on how to set the integrity level of the pipe to Untrusted while creating the pipe. It’s also a good example of how to create pipe applications that will be using Anonymous pipes on Vista onwards systems:
=> Blog link: (in German)
http://blog.m-ri.de/index.php/2009/12/08/windows-integrity-control-schreibzugriff-auf-eine-named-pipe-eines-services-ueber-anonymen-zugriff-auf-vista-windows-2008-server-und-windows-7/
Note: It's a 3rd party link so please connect to it at your own risk.
=> Just a few notes from the source code to further explain how it could be done:
a) While the pipe is created, a security attributes structure is passed:
hPipe = CreateNamedPipe(
server,
PIPE_ACCESS_DUPLEX,
PIPE_TYPE_MESSAGE | PIPE_READMODE_MESSAGE | PIPE_WAIT,
PIPE_UNLIMITED_INSTANCES,
sizeof(DWORD),
0,
NMPWAIT_USE_DEFAULT_WAIT,
&sa );
b) Especially integrity level related part of that the security attribute structure is built as follows:
...
// We need this only if we have Windows Vista, Windows 7 or Windows 2008 Server
OSVERSIONINFO osvi;
osvi.dwOSVersionInfoSize = sizeof(osvi);
if (!GetVersionEx(&osvi))
{
DisplayError( L"GetVersionInfoEx" );
return FALSE;
}
// If Vista, Server 2008, or Windows7!
if (osvi.dwMajorVersion>=6)
// Now the trick with the SACL:
// We set SECURITY_MANDATORY_UNTRUSTED_RID to SYSTEM_MANDATORY_POLICY_NO_WRITE_UP
// Anonymous access is untrusted, and this process runs equal or above medium
// integrity level. Setting "S:(ML;;NW;;;LW)" is not sufficient.
_tcscat(szBuff,_T("S:(ML;;NW;;;S-1-16-0)"));
The highlighted part will cause the integrity level to be set to Untrusted on the pipe object while the pipe is created via CreateNamedPipe().
You can find more information on Integrity check at the following link:
http://msdn.microsoft.com/en-us/library/bb625963.aspx Windows Integrity Mechanism Design
http://msdn.microsoft.com/en-us/library/aa379588(VS.85).aspx SetSecurityInfo Function
http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx CreateFile Function
Thanks,
Murat
Hi there, In today's blog, I would like to talk about NLB cluster access problems that our customers experience most of the time... When Microsoft NLB cluster operates in multicast mode, in certain scenarios you may not be able to access the NLB cluster IP address from remote subnets whereas suame subnet access keeps working fine. You can find more information at the two most common scenarios below:
In today's blog, I would like to talk about NLB cluster access problems that our customers experience most of the time...
When Microsoft NLB cluster operates in multicast mode, in certain scenarios you may not be able to access the NLB cluster IP address from remote subnets whereas suame subnet access keeps working fine. You can find more information at the two most common scenarios below:
Problem 1:
When NLB cluster on Windows 2008/SP2 operates in multicast mode, due to a problem with NLB implementation on 2008 remote subnets cannot access NLB cluster IP address
Solution 1:
- This problem was stemming from NLB implementation
- This has been fixed by Microsoft with the hotfix KB960916.
- KB960916 is already included in Windows 2008 SP2
Problem 2:
When NLB cluster on Windows 2003 or Windows 2008 operates in multicast mode, remote subnets cannot access NLB cluster IP address. That second problem stems from the fact that some vendors (like Cisco) don't accept mapping L3 unicast IP addresses to L2 multicast MAC addresses (this happens when NLB cluster operates in multicast mode - L3 unicast IP address is the NLB cluster IP address and L2 mac address is the multicast MAC address that is chose by NLB) so you have to create a static mapping on the router to avoid such a problem. You can find more information about this problem at the following link:
http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a0080a07203.shtml
(Taken from the above link)
Multicast Mode
Another solution is to use multicast mode in MS NLB configuration GUI instead of Unicast mode. In Multicast Mode, the system admin clicks the IGMP Multicast button in the MS NLB configuration GUI. This choice instructs the cluster members to respond to ARPs for their virtual address using a multicast MAC address for example 0300.5e11.1111 and to send IGMP Membership Report packets. If IGMP snooping is enabled on the local switch, it snoops the IGMP packets that pass through it. In this way, when a client ARPs for the cluster’s virtual IP address, the cluster responds with multicast MAC for example 0300.5e11.1111. When the client sends the packet to 0300.5e11.1111, the local switch forwards the packet out each of the ports connected to the cluster members. In this case, there is no chance of flooding the ARP packet out of all the ports. The issue with the multicast mode is virtual IP address becomes unreachable when accessed from outside the local subnet because Cisco devices do not accept an arp reply for a unicast IP address that contains a multicast MAC address. So the MAC portion of the ARP entry shows as incomplete. (Issue the command show arp to view the output.) As there is no MAC portion in the arp reply, the ARP entry never appeared in the ARP table. It eventually quit ARPing and returned an ICMP Host unreachable to the clients. In order to override this, use static ARP entry to populate the ARP table as given below. In theory, this allows the Cisco device to populate its mac-address-table. For example, if the virtual ip address is 172.16.63.241 and multicast mac address is 0300.5e11.1111, use this command in order to populate the ARP table statically:
Solution 2:
In order to resolve that problem, you have two choices:
a) Adding a static ARP entry on the router
b) Changing NLB cluster mode to Unicast
Also please always keep in mind the following when troubleshooting NLB problems:
1) Do I run the latest NLB driver available from Microsoft? We have released a few updates on NLB drivers on Windows 2003, Windows 2008 and Windows 2008 R2 to address a few problems
2) Do I run the latest NIC driver and teaming driver? We generally prefer not to run teaming on NLB clusters and may ask to dissolve the teaming if needed even though we don't have strict "not supported" statement.
3) Do the NLB rules are correctly configured? The most common problem with that is to set affinity to "None" for stateful protocols which causes many NLB cluster access problems.
4) Do I run the latest TCPIP driver? (preferrably the latest security update which updates TCPIP driver)
5) Do I run the latest 3rd party filter drivers that run at NDIS layer? (for example security drivers)
6) If NLB cluster runs on Windows 2008 R2 Hyper-V, do you disable "Enable spoofing of MAC addresses"?
I'm going to talk about troubleshooting approaches in another blog post.