Brad Rutkowski's Blog : Networking

Taking a circular netmon capture from the command prompt

Brad Rutkowski — Fri, 22 Feb 2008 22:23:42 GMT

You've probably heard that netmon3.1 is out, but you might not know that you can easily launch a capture at the command prompt. I find this useful when we're waiting on a repro, we want a capture, but we don’t know when that's going to happen. Sure you could set this up in the GUI too, but who wants to do that when it's as easy as this?

The below will setup a capture on all networks that the system is attached to and wait until I hit ctrl+c (you can see its been a while with no repro). The CHN extension used tells netmon to take multiple captures in a chain (see file syntax). I also put some examples at the bottom so you can see what else you can do. Have fun!

C:\Windows\system32>nmcap /capture /network * /File netmoncap.chn:100M
Netmon Command Line Capture (nmcap) 03.01.0512.0000

Saving info to:
C:\Windows\system32\netmoncap.cap - using chain captures of size 100.00 MB.

ATTENTION: Conversations Enabled: consumes more memory (see Help for details)

Exit by Ctrl+C

Saved Frames: 9232127 Capture Frames: 9438779 (44181 seconds)

Hit Ctrl+C

Cancelled by user

Final Results : Saved Frames: 722 Capture Frames: 722

C:\Program Files\Microsoft Network Monitor 3>dir netmoncap.cap
Volume in drive C has no label.
Volume Serial Number is FCC3-5AF7

Directory of C:\Program Files\Microsoft Network Monitor 3

02/22/2008 09:06 AM           384,748 netmoncap.cap
               1 File(s)        384,748 bytes
               0 Dir(s) 16,699,654,144 bytes free

Here's the breakdown fo the /File syntax:

/File <Capture File>[:<File Size Limit>]
    Name of capture file to save frames to. Extensions are used to determine
    the behavior of nmcap.
     .cap -- Netmon 2 capture file
     .chn -- Series of Netmon 2 capture files: t.cap, t(1).cap, t(2).cap...
    <File Size Limit> are optional. It limits the file size of each capture
    file generated. Default single capture file size limit is 20M. The
    upper bound of the file size limit is 500M. The lower bound of the file
    size limit depends on the frame size captured. (Note that the maximal size
    of ethernet frames is 1500 Bytes)
    The files are circular, so once the size limit is reached, new data will
    overwrite older data.
    Example Usage: /File t.cap:50M

Some other examples from the NMCAP help:

This example starts capturing network frames that DO NOT contain ARPs, ICMP,
NBtNs and BROWSER frames. If you want to stop capturing, Press Control+C.

nmcap /network * /capture (!ARP AND !ICMP AND !NBTNS AND !BROWSER) /File NoNoise.cap

Starts capturing network frames immediately. All TCP frames that have a source
port or destination port of 80 are saved to the chained capture files named
test.cap, test(1).cap, test(2).cap, ... When the user presses the 'x' key the
program stops.

nmcap /network * /capture tcp.port == 80 /file c:\temp\test.chn:6M /stopwhen /keypress x

This example starts capturing network frames that are TCP Continuations. The
capture filter is searching for String "Continuation in TCP Frame Summary
Description. In order to see the complete list of Netmon Properties that are
filterable,type ".Property" in the Netmon Filter UI.

nmcap /network * /capture contains(.Property.Description, \"Continuation\") /File TCPContinuations.cap

Technorati Tags: Netmon,Network Monitor

Domain not available when trying to TS onto a Windows 2003 server.

Brad Rutkowski — Thu, 16 Aug 2007 20:00:00 GMT

Issue came in this week where when you attempted to logon to a server it would not authenticate your request and would give you a message indicating the "domain is not available". If you tried logging on via your UPN, then it would give a slightly different error message indicating that "there is not enough storage to complete this operation".

After ruling out DNS and routing, I had the person run nltest /sc_query:BRADFOREST to see what DC it was pointing at and found that it did not have a secure channel to a DC which might be a reason we can't authenticate to the server. :) When we tried to reset the secure channel it would fail with error code 8 (ERROR_NOT_ENOUGH_MEMORY).) So we cranked up netlogon debug logging and then I repro'd the issue again. We could then see this in the netlogon debug log:

08/14 22:55:06 [SESSION] BRADFOREST: NlSetServerClientSession: New DC is an NT 5 DC: \\brad-dc-01.bradforest.local
08/14 22:55:06 [SESSION] BRADFOREST: NlSetServerClientSession: New DC is in closest site: \\brad-dc-01.bradforest.local
08/14 22:55:06 [SESSION] BRADFOREST: NlSetServerClientSession: New DC runs the time service: \\brad-dc-01.bradforest.local
08/14 22:55:06 [SESSION] BRADFOREST: NlSetServerClientSession: New discovery flags: 0x1dc; Old flags: 0x0
08/14 22:55:06 [SESSION] BRADFOREST: NlDiscoverDc: Found DC \\brad-dc-01.bradforest.local
08/14 22:55:06 [SESSION] BRADFOREST: NlStartApiClientSession: Bind to server \\brad-dc-01.bradforest.local (TCP) 0 (Retry: 0).
08/14 22:55:06 [MAILSLOT] Going to wait on mailslot. (Timeout: 45000)
08/14 22:55:06 [CRITICAL] NlPrintRpcDebug: Dumping extended error for I_NetServerReqChallenge with 0xc0000017
08/14 22:55:06 [CRITICAL] [0] ProcessID is 780 <-------------------------LSASS.exe
08/14 22:55:06 [CRITICAL] [0] System Time is: 8/14/2007 21:55:6:372
08/14 22:55:06 [CRITICAL] [0] Generating component is 8
08/14 22:55:06 [CRITICAL] [0] Status is 14
08/14 22:55:06 [CRITICAL] [0] Detection location is 313
08/14 22:55:06 [CRITICAL] [0] Flags is 0
08/14 22:55:06 [CRITICAL] [0] NumberOfParameters is 0
08/14 22:55:06 [CRITICAL] [1] ProcessID is 780
08/14 22:55:06 [CRITICAL] [1] System Time is: 8/14/2007 21:55:6:372
08/14 22:55:06 [CRITICAL] [1] Generating component is 8
08/14 22:55:06 [CRITICAL] [1] Status is 10055
08/14 22:55:06 [CRITICAL] [1] Detection location is 311
08/14 22:55:06 [CRITICAL] [1] Flags is 0
08/14 22:55:06 [CRITICAL] [1] NumberOfParameters is 3
08/14 22:55:06 [CRITICAL]      Long val: 1025
08/14 22:55:06 [CRITICAL]      Pointer val: 0
08/14 22:55:06 [CRITICAL]      Pointer val: 0
08/14 22:55:06 [CRITICAL] [2] ProcessID is 780
08/14 22:55:06 [CRITICAL] [2] System Time is: 8/14/2007 21:55:6:372
08/14 22:55:06 [CRITICAL] [2] Generating component is 8
08/14 22:55:06 [CRITICAL] [2] Status is 10055
08/14 22:55:06 [CRITICAL] [2] Detection location is 315
08/14 22:55:06 [CRITICAL] [2] Flags is 0
08/14 22:55:06 [CRITICAL] [2] NumberOfParameters is 0
08/14 22:55:06 [CRITICAL] BRADFOREST: NlSessionSetup: Session setup: cannot I_NetServerReqChallenge 0xc0000017
08/14 22:55:06 [MISC] Eventlog: 5719 (1) "BRADFOREST" 0xc0000017 c0000017   ....

Some interesting things to look at, first off what is 0xc0000017? Well we can use err.exe to see what that translates to.

C:\Windows\system32>err 0xc0000017
# for hex 0xc0000017 / decimal -1073741801
STATUS_NO_MEMORY
# {Not Enough Quota}
# Not enough virtual memory or paging file quota is available
# to complete the specified operation.

Well that pretty much flies with what I was seeing when trying to logon via UPN. We can also see two status codes being returned during the secure channel setup: 14 and 10055.

C:\Windows\system32>err /winerror.h 14
# winerror.h selected.
# for decimal 14 / hex 0xe
ERROR_OUTOFMEMORY
# Not enough storage is available to complete this operation. <-- This is what I was getting when trying to TS via UPN.

C:\Windows\system32>err /winerror.h 10055
# winerror.h selected.
# for decimal 10055 / hex 0x2747
WSAENOBUFS <--------------------HMMMMM?
# An operation on a socket could not be performed because the
# system lacked sufficient buffer space or because a queue
# was full.

So now that is interesting, so the next thing I did was do a netstat -s and looked at the statistics of ports and didn't see anything obvious and I then added the handles column in task manager and noticed that their custom application had 17,000 handles open. Turns out that most of those handles were outgoing calls and used up all the ephemeral ports. We had to set the MAXUSERPORT value in the registry to allow more ports to be used, once we did that everything returned to normal.

Ephemeral Ports

The number of user-accessible ephemeral ports that can be used to source outbound connections is configurable using the MaxUserPorts registry parameter. By default, when an application requests any socket from the system to use for an outbound call, a port between the values of 1024 and 5000 is supplied. The MaxUserPorts parameter can be used to set the value of the uppermost port that the administrator chooses to allow for outbound connections. For instance, setting this value to 10,000 (decimal) would make approximately 9000 user ports available for outbound connections.

Here is the KB article for the issue: http://support.microsoft.com/kb/196271

Here you can read about another setting called TCP TIME-WAIT delay which is how long the port hangs around before being terminated completely (4 minutes). This can also cause issues with apps that perform many outbound connections in a short time may use up all available ports before the ports can be recycled.

Technorati tags: Windows 2003, Networking

How to know if TCP offload is working

Brad Rutkowski — Fri, 10 Aug 2007 23:10:00 GMT

So you went out and got yourself a new server and it came with TOE functionality, and now you're playing Windows 2008 which has TCP offload enabled but you just want to know if its actually offloading traffic. Here's the only way I know of finding what traffic is offloaded without setting breakpoints in the debugger.

First off to check if TCP offload is enabled:

C:\>netsh int tcp show global
Querying active state...

TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State          : enabled
Chimney Offload State               : enabled <-----
Receive Window Auto-Tuning Level    : normal
Add-On Congestion Control Provider : ctcp
ECN Capability                      : disabled
RFC 1323 Timestamps                 : disabled

To turn it on/off (does not require a reboot)

netsh int tcp set global chimney=disabled

netsh int tcp set global chimney=enabled

So how do we see if traffic is offloaded? You run netstat -nt, the 't' dumps their current offload state. I used findstr just to grab the offloaded connections.

C:\>netstat -nt | findstr /i offloaded
TCP    110.100.44.52:445      10.5.17.2:1369     ESTABLISHED     Offloaded
TCP    10.100.44.52:445       1.56.15.14:4741    ESTABLISHED     Offloaded
TCP    10.100.44.52:49157     1.198.5.2:2444     ESTABLISHED     Offloaded
TCP    10.100.44.52:49157     1.100.4.219:2255   ESTABLISHED     Offloaded
TCP    10.100.44.52:49157     1.58.6.50:54620    ESTABLISHED     Offloaded
TCP    10.100.44.52:49157     1.58.20.40:50442   ESTABLISHED     Offloaded
TCP    10.100.44.52:49157     1.58.25.15:1191    ESTABLISHED     Offloaded
TCP    10.100.44.52:49157     1.148.8.6:58308    ESTABLISHED     Offloaded
TCP    10.100.44.52:49449     1.10.3.2:1025      ESTABLISHED     Offloaded

UPDATE:

Windows 2003 its a bit different:

Netsh int ip set chimney DISABLED

Netsh int ip set chimney ENABLED

Want to know more about Scalable Networking?

http://technet.microsoft.com/en-us/network/bb545631.aspx

Net Send in Windows Vista

Brad Rutkowski — Mon, 06 Aug 2007 19:45:51 GMT

You might find yourself looking for "net send" in Vista and wonder where it went? Before you flip out, there is a solution. Use MSG.exe

C:\Windows\system32>msg /?
Send a message to a user.

MSG {username | sessionname | sessionid | @filename | *}
[/SERVER:servername] [/TIME:seconds] [/V] [/W] [message]

username            Identifies the specified username.
sessionname         The name of the session.
sessionid           The ID of the session.
@filename           Identifies a file containing a list of usernames,
                      sessionnames, and sessionids to send the message to.
*                   Send message to all sessions on specified server.
/SERVER:servername server to contact (default is current).
/TIME:seconds       Time delay to wait for receiver to acknowledge msg.
/V                  Display information about actions being performed.
/W                  Wait for response from user, useful with /V.
message             Message to send. If none specified, prompts for it
                      or reads from stdin.

C:\msg /server:brad-dc-01 console "You're machine needs to be rebooted."