Hi there,

In a number of support cases, I was requested to take a look at "access denied" network traces. In some cases the reason of why we get the access denied is related to Kerberos authentication and Local system account so I wanted to touch this subject in today's blog.

A service or application might be configured to run under "Local System" security context. Normally when such a service or application (like a scheduled task) needs to access network resources, it uses "domain\computername$" account to authenticate itself to remote parties. In one of the scenarios I was involved in, a scheduled task (which runs under Local system security context) ran a script which in turn accessed a server share (\\server\share). When the script was run that way, it got an access denied error from the destination server and failed to run properly.

In the network trace captured when that failure occured, it was seen that the source client tried to authenticate against the remote server with NULL credentials. Since anonymous authentication was not enabled at the destination server, it didn't authenticate the client and returned ACCESS DENIED:

Source            Destination    Protocol Info
======            ========    =======  ==== 
10.1.1.1          10.1.1.2          SMB   Session Setup AndX Request, NTLMSSP_NEGOTIATE
10.1.1.2          10.1.1.1          SMB   Session Setup AndX Response, NTLMSSP_CHALLENGE, Error: STATUS_MORE_PROCESSING_REQUIRED
10.1.1.1          10.1.1.2          SMB   Session Setup AndX Request, NTLMSSP_AUTH, User: \
10.1.1.2          10.1.1.1          SMB   Session Setup AndX Response
10.1.1.1          10.1.1.2          SMB   Tree Connect AndX Request, Path: \\Server\Share
10.1.1.2          10.1.1.1          SMB   Tree Connect AndX Response, Error: STATUS_ACCESS_DENIED
10.1.1.1          10.1.1.2          SMB   Logoff AndX Request
10.1.1.2          10.1.1.1          SMB   Logoff AndX Response

Note: It can be seen in the 3rd frame that the client is sending NULL credentials

The above problem was stemming from a Kerberos authentication failure. Normally "domain\computername$" account is used when the service/application runs under Local System security context but it's done only if the authentication protocol is Kerberos. So if Kerberos works fine, than the given account is used. If Kerberos authentication fails for a reason (like name resolution issues etc) and the client needs to fall back to NTLM, NULL credentials are used instead of "domain\computername$".

If the related service/application is configured to run under "Network service" account context, domain\computername$ account is used when authenticating with either Kerberos or NTLM protocol.

In the above scenario, after addressing the Kerberos issue, the NULL credentials problem was resolved.

As an additional information for Windows 7 & Windows 2008 R2, an application/service running under "Local System account" security context can use domain\computername$ account when authenticating against remote servers with either Kerberos or NTLM authentication. This could be configured via group policies for those systems

http://technet.microsoft.com/en-us/library/dd566199(WS.10).aspx Changes in NTLM Authentication

Hope this helps

Thanks,
Murat

Hi there,

In one of the past cases, one of our customers wanted to know if we supported policy based routing on Windows 2003 or later OSes. First of all, it might be useful to clarify what "policy based routing" means in this context. Let's take the following as an example:

"A server is running as a router and have 3 network interfaces. When the server receives a packet from a specific host (let’s say running at a certain IP address) from one of its interfaces (say interface1), we would like the server to always route that host’s packets through interface2 without consulting the routing table. (The criteria might be different for different scenarios such as "all packets with a destination of TCP port 80 to be sent out from interface3 etc)"

This kind of advanced routing decisions are generally supported by network hardware vendors like Cisco. For example, by using route-map configuration in Cisco IOS, you can affect the conventional routing decisions made by looking up the routing table. You can find more information on that at the following link:

http://www.cisco.com/en/US/docs/ios/12_0/qos/configuration/guide/qcpolicy.html Configuring Policy-Based Routing

And the answer to the original question is: No, we don't support policy based routing on Windows server OSes since this is generally a feature that would be needed on hardware routers whose main purpose is to do packet routing.

Hope this helps

Thanks,
Murat

Hi there,

Recently I dealt with a connectivity issue that occurs when deploying OS images with Windows 2008 R2 WDS server to PXE clients running in different subnets. As you may already know, we have a workaround for router MTU size incompatibilities seen when deploying OS images to remote subnets in Windows 2008:

http://support.microsoft.com/kb/975710 Operating system deployment over a network by using WDS fails in Windows Server 2008

The same router packet drop issue was present in my case. Even we configured the MaximumBlockSize reg key on the Windows 2008 R2 WDS server, the issue was still in place and the TFTP server (part of WDS server) was still sending data in chunks requested by the client rather than the size we tried to limit wit the MaximumBlockSize reg key:

Note:
10.1.1.1 is PXE client
10.2.2.2 is TFTP server

=> Client is asking for a TFTP block size of 1456 bytes and TFTP server (that’s running as a part of WDS server) is honoring that request:

10.1.1.1          10.2.2.2            TFTP        Read Request, File: boot\x86\wdsnbp.com\000, Transfer type: octet\000, blksize\000=1456\000
10.2.2.2          10.1.1.1            TFTP        Option Acknowledgement, blksize\000=1456\000

=> Then TFTP server begins sending the first 1456 bytes block of wdsnbp.com file:

10.2.2.2            10.1.1.1          TFTP        Data Packet, Block: 1

=> Most likely because the router drops that packet, TFTP client doesn’t receive it and hence it re-sends the Read file request once more:

10.1.1.1          10.2.2.2            TFTP        Read Request, File: boot\x86\wdsnbp.com\000, Transfer type: octet\000, blksize\000=1456\000

=> And TFTP server sends the same block once more and tries a number of times:

10.2.2.2         10.1.1.1      TFTP      Data Packet, Block: 1
10.2.2.2         10.1.1.1      TFTP      Data Packet, Block: 1
10.2.2.2         10.1.1.1      TFTP      Data Packet, Block: 1
10.2.2.2         10.1.1.1      TFTP      Data Packet, Block: 1
...

So PXE boot fails at the end.

The analysis showed us that the reg key didn't take effect on Windows 2008 R2 WDS server. After a source code review, I saw that the feature wasn't integrated into Windows 2008 R2 RTM code due to release timing of Windows 2008 R2 and the fix for Windows 2008. I verified with internal resources and can say that the feature will be a part of Windows 2008 R2 SP1. The SP1 public beta is being made available by the end of July:

http://blogs.technet.com/b/itproaustralia/archive/2010/06/08/windows-7-and-windows-server-2008-r2-sp1-beta-available-end-of-july.aspx  Windows 7 and Windows Server 2008 R2 SP1 Beta available end of July

Hope this helps

Thanks,
Murat

Hi,

I had to deal with a number of support cases where IPReassemblytimeout reg key was set but didn't take effect on Windows 2003 or a later system and I thought I should be sharing more information about this here. Here are some details:

IP fragmentation is needed when an upper layer packet whose payload is bigger than the IP MTU needs to be sent to a destination. This could happen when the packet initially leaves the host or could happen when a router needs to forward a packet that it received through one interface (with a bigger MTU) to another interface (Smaller MTU). That also happens when packets need to traverse VPN links where there's an additional VPN related overhead causing the original packet to be fragmented.

The final receiver of the fragmented IP packets re-assembles those fragments and forms the original packet before passing to upper layer protocols. Receiver waits for a period called "re-assembly timeout" for all the fragments that belong to the original packet to be received. If any one of the fragments is dropped on the way, receiver drops the other fragments belonging to the original packet.

In NT 3.1, there was a registry key called "IPReassemblytimeout" (which is referred to by KB 102973). But that registry key doesn't apply to Windows 2000, XP, 2003, Vista, 2008, Windows 7 or 2008 R2!

Some more facts:

1) IP Re-assembly timeout is hardcoded on Windows 2000, Windows 2003, Windows XP, Windows Vista, Windows 2008, Windows 7 and Windows 2008 R2 and cannot be changed by any means (registry, netsh etc)

2) For Windows Vista, Windows 2008, Windows 7 and Windows 2008 R2, it's hardcoded to 60 seconds as per section 4.5 of RFC 2460:

"If insufficient fragments are received to complete reassembly of a packet within 60 seconds of the reception of the first-arriving fragment of that packet, reassembly of that packet must be abandoned and all the fragments that have been received for that packet must be discarded."

For more information, please see RFC2460 (http://www.ietf.org/rfc/rfc2460.txt)

3) For Windows 2000, Windows XP and Windows 2003, it's at least 60 seconds but it may be higher depending on the value of TTL in the IP header and it may go up to 120 seconds.

Hope this helps..

Thanks,
Murat

Hi,

In this blog entry, I will be discussing TCP keepalive mechanism and will also provide some information about configuration options on Windows systems.

a) Definition

Let's first understand the mechanism. A TCP keep-alive packet is simply an ACK with the sequence number set to one less than the current sequence number for the connection. A host receiving one of these ACKs responds with an ACK for the current sequence number. Keep-alives can be used to verify that the computer at the remote end of a connection is still available. TCP keep-alives can be sent once every KeepAliveTime (defaults to 7,200,000 milliseconds or two hours) if no other data or higher-level keep-alives have been carried over the TCP connection. If there is no response to a keep-alive, it is repeated once every KeepAliveInterval seconds. KeepAliveInterval defaults to 1 second. NetBT connections, such as those used by other Microsoft networking components, send NetBIOS keep-alives more frequently, so normally no TCP keep-alives are sent on a NetBIOS connection. TCP keep-alives are disabled by default, but Windows Sockets applications can use the setsockopt() function to enable them.

b) Configuration

Now let's talk a little bit about configuration options. There're 3 registry keys where you can affect TCP Keepalive mechanism on Windows systems:

KeepAliveInterval
Key: Tcpip\Parameters
Value Type: REG_DWORD-time in milliseconds
Valid Range: 1-0xFFFFFFFF
Default: 1000 (one second)
Description: This parameter determines the interval between TCP keep-alive retransmissions until a response is received. Once a response is received, the delay until the next keep-alive transmission is again controlled by the value of KeepAliveTime. The connection is aborted after the number of retransmissions specified by TcpMaxDataRetransmissions have gone unanswered.

Notes:
TCPIP driver waits for a TCP Keepalive ACK for the duration of time specified in this registry entry.

KeepAliveTime
Key: Tcpip\Parameters
Value Type: REG_DWORD-time in milliseconds
Valid Range: 1-0xFFFFFFFF
Default: 7,200,000 (two hours)
Description: The parameter controls how often TCP attempts to verify that an idle connection is still intact by sending a keep-alive packet. If the remote system is still reachable and functioning, it acknowledges the keep-alive transmission. Keep-alive packets are not sent by default. This feature may be enabled on a connection by an application

Notes:
In order for a TCP session to stay idle, there should be no data sent or received.

c) If OS is Windows XP/2003 the following registry entry applies:

TcpMaxDataRetransmissions
Key: Tcpip\Parameters
Value Type: REG_DWORD-number
Valid Range: 0-0xFFFFFFFF
Default: 5
Description: This parameter controls the number of times that TCP retransmits an individual data segment (not connection request segments) before aborting the connection. The retransmission time-out is doubled with each successive retransmission on a connection. It is reset when responses resume. The Retransmission Timeout (RTO) value is dynamically adjusted, using the historical measured round-trip time (Smoothed Round Trip Time) on each connection. The starting RTO on a new connection is controlled by the TcpInitialRtt registry value.

Notes:
This registry entry determines the number of TCP retransmissions for an individual TCP segment. There's no special registry entry to determine the retransmission behavior of TCP Keepalives and this registry entry is also used for the TCP keepalive scenario.

Important note: If OS is Windows Vista/2008, the number of TCP Keepalive attempts are hardcoded to 10 and could not be adjusted via the registry.

d) Some special considerations

=> Even if TCP KeepaliveTime and TCPKeepAliveInterval registry keys are set to a specific value (TCPIP driver uses the deafult values even if we don't set these registry keys from the registry), TCPIP driver won't start sending TCP Keepalives until Keepalives are enabled via various methods at upper layers (layers above TCPIP driver).

=> Native Socket applications can enable TCP keepalives by using anyone of the following methods:

- setsockopt() with SO_KEEPALIVE option
- WSAIoctl() with SIO_KEEPALIVE_VALS option (it's also possible to change Keepalive timers with this API call dynamically on a per-socket basis)

=> Managed applications (.NET), can use one of the following methods:

- SetSocketOption method from Socket Class in System.Net.Sockets namespace
- GetSocketOption method from Socket Class in System.Net.Sockets namespace

=> Effect of using Keepalives on bandwidth usage

Since TCP Keepalives are TCP segments without data (and the SEQ number set to one less than the current SEQ number), Keepalive usage bandwidth usage can simply be neglected. There's an example below to give an idea about how big a TCP Keepalive packet could be:

- 14 bytes (L2 header - Assuming that Ethernet protocol is used. This could be even lower for other WAN protocols like PPP/HDLC/etc)
- 20 bytes (IP header - assuming no IP options are used)
- 20 bytes (TCP header - assuming no TCP options are used)

Total: ~54 bytes

Even if TCP Keepalive interval is set to 5 minutes or so (default is 2 hours), given that TCP connection goes idle, TCPIP driver will send a ~54 TCP Keepalive message every 5 minutes and as can be seen it could simply be neglected.

You may also find some references below:

References
=============================
http://www.microsoft.com/downloads/details.aspx?FamilyID=06c60bfe-4d37-4f50-8587-8b68d32fa6ee&displaylang=en
Microsoft Windows Server 2003 TCP/IP Implementation Details

http://www.microsoft.com/downloads/details.aspx?FamilyId=12AC9780-17B5-480C-AEF7-5C0BDE9060B0&displaylang=en
TCP/IP Registry Values for Microsoft Windows Vista and Windows Server 2008

http://msdn.microsoft.com/en-us/library/ms740476(VS.85).aspx setsockopt Function
http://msdn.microsoft.com/en-us/library/ms741621(VS.85).aspx WSAIoctl Function
http://msdn.microsoft.com/en-us/library/system.net.sockets.socket.setsocketoption.aspx Socket..::.SetSocketOption Method
http://msdn.microsoft.com/en-us/library/system.net.sockets.socket.getsocketoption.aspx Socket..::.GetSocketOption Method

Hope this helps.

Thanks,
Murat